Worker & serving

On this page

The Worker builder, the transports, and what a worker declares about itself.

record AttachOptionSpec

source

public record AttachOptionSpec( String name, String description, Field valueField, FieldVector defaultVector, boolean required)

Description

Declares an ATTACH-time option this worker accepts. Users supply a value at ATTACH time (e.g. ATTACH '…' AS x (TYPE vgi, opt_int 42)); the resolved value flows to catalog_attach via CatalogAttachRequest.options as a one-row record batch keyed by option name. The wire format is identical to SettingSpec.

defaultVector is a length-1 Arrow vector pre-materialised at registration time. Keeping the default in vector form lets the merge path (catalog_attach) copy it via TransferPair alongside user-supplied values — uniform, no type dispatch on the hot path. Allocate the spec once at worker startup; the vector is owned by the spec and freed when the process exits.

Members

AttachOptionSpec

Rejects the contradictory required-plus-default combination.

AttachOptionSpec of(String name, String description, ArrowType type, Object defaultValue)

Convenience: scalar option with a Java-valued default.

AttachOptionSpec required(String name, String description, ArrowType type)

Convenience: an option the caller must supply at ATTACH time.

A catalog that cannot be attached without this option advertises that at discovery, so a client can say so before attempting the attach rather than surfacing a failure that reads like an empty catalog.

AttachOptionSpec of(String name, String description, ArrowType type, List<Field> children, Object defaultValue)

Convenience: complex option (list/struct) with children + default.

ArrowType type()

the option's Arrow value type, read from {@link #valueField}

List<Field> children()

the value field's child fields (empty for flat scalar options)

record CatalogDataVersionRelease

source

public record CatalogDataVersionRelease( String version, Instant releasedAt, String summary, String notesUrl)

Description

One published data version of a catalog, surfaced through catalog_catalogs() so clients can render a release timeline before attaching. Mirrors vgi-python CatalogDataVersionRelease.

The CatalogInfo.releases list this belongs to is ordered newest-first, with at most one entry per version.

Members

CatalogDataVersionRelease

Enforces the wire schema’s nullability: version and releasedAt must be present, and a null summary is normalised to the empty string (notesUrl alone may stay null).

record ExtraCatalog

source

public record ExtraCatalog(String name, String implementationVersion, String dataVersion, String schemaComment, List<AttachOptionSpec> attachOptions)

Description

An auxiliary catalog served by the same worker process next to the main catalog, MetaWorker-style: it appears as its own row in catalog_catalogs(), attaches by name with its own versions and a random per-ATTACH opaque id, and owns the functions registered into it through the registerExtraCatalog* methods (those functions are listed only under this catalog’s attaches, and hidden from the main catalog’s).

Members

ExtraCatalog

Defensive copy; a null option list reads as none declared.

ExtraCatalog(String name, String implementationVersion, String dataVersion, String schemaComment)

An auxiliary catalog declaring no attach options of its own.

Worker registerExtraCatalog(ExtraCatalog catalog)

Register an auxiliary catalog served next to the main one.

Map<String, ExtraCatalog> extraCatalogs()

The auxiliary catalogs registered via #registerExtraCatalog.

Worker registerExtraCatalogTable(String catalogName, CatalogTable t)

Register a catalog table owned by an auxiliary catalog. Such tables are enumerated only under that catalog’s attaches (and never appear in the main catalog’s listings). The scan functions they reference should be registered through #registerExtraCatalogTableFunction into the same catalog so they are likewise owned by it.

Map<String, List<CatalogTable>> extraCatalogTables()

Catalog tables owned by auxiliary catalogs, keyed by catalog name.

String schemaOf(Object fn)

The catalog schema fn is declared in — the schema DuckDB registers it into and therefore the one a bind request names. Every registered function has exactly one: a registration that names no schema resolves to #defaultSchema(), which is a real home, not a wildcard. Nothing is visible in more than one schema.

String catalogOf(Object fn)

The auxiliary catalog fn is declared in, or null when it belongs to this worker’s own catalog. Ownership is always explicit — a function is registered into exactly one catalog — so two auxiliary catalogs may declare the very same function name and still dispatch apart.

Worker registerScalar(ScalarFunction fn)

Register a scalar function, callable from SQL and enumerated through catalog_schema_contents_functions.

Worker registerScalar(String schemaName, ScalarFunction fn)

Register a scalar function into a named schema of this worker’s catalog (rather than #defaultSchema()). The same function name may be registered in more than one schema: DuckDB registers one entry per schema, and bind requests carry the schema so each call reaches the implementation the caller named.

Worker registerTable(String schemaName, TableFunction fn)

Register a table function into a named schema of this worker’s catalog (rather than #defaultSchema()). See #registerScalar(String, ScalarFunction).

Worker registerTableInOut(String schemaName, TableInOutFunction fn)

Register a table-in-out function into a named schema of this worker’s catalog (rather than #defaultSchema()). See #registerScalar(String, ScalarFunction).

Worker registerTableBuffering(String schemaName, farm.query.vgi.buffering.TableBufferingFunction fn)

Register a table-buffering function into a named schema of this worker’s catalog (rather than #defaultSchema()). See #registerScalar(String, ScalarFunction).

Worker registerAggregate(String schemaName, AggregateFunction<?> fn)

Register an aggregate function into a named schema of this worker’s catalog (rather than #defaultSchema()). See #registerScalar(String, ScalarFunction).

Worker registerExtraCatalogScalar(String catalogName, String schemaName, ScalarFunction fn)

Register a scalar function owned by an auxiliary catalog, in a named schema of it. The function is listed only under that catalog’s attaches and hidden from the main catalog’s. Ownership is explicit per function, so two auxiliary catalogs can declare the SAME function name and still dispatch apart — the attach names the catalog.

Worker registerExtraCatalogTableFunction(String catalogName, String schemaName, TableFunction fn)

Register a table function owned by an auxiliary catalog, in a named schema of it. See #registerExtraCatalogScalar.

Worker registerExtraCatalogTableInOut(String catalogName, String schemaName, TableInOutFunction fn)

Register a table-in-out function owned by an auxiliary catalog, in a named schema of it. See #registerExtraCatalogScalar.

Worker registerExtraCatalogTableBuffering(String catalogName, String schemaName, farm.query.vgi.buffering.TableBufferingFunction fn)

Register a table-buffering function owned by an auxiliary catalog, in a named schema of it. See #registerExtraCatalogScalar.

Worker registerTable(TableFunction fn)

Register a table function, callable from SQL and enumerated through catalog_schema_contents_functions.

Worker registerAggregate(AggregateFunction<?> fn)

Register an aggregate function, callable from SQL and enumerated through catalog_schema_contents_functions.

Worker registerTableInOut(TableInOutFunction fn)

Register a table-in-out function (consumes an input relation, streams an output relation), enumerated through catalog_schema_contents_functions.

Worker registerScalars(Iterable<? extends ScalarFunction> fns)

Register several scalar functions; equivalent to calling #registerScalar(ScalarFunction) for each.

Worker registerTables(Iterable<? extends TableFunction> fns)

Register several table functions; equivalent to calling #registerTable(TableFunction) for each.

Worker registerUnlistedTable(TableFunction fn)

Register a table function that is dispatchable but not advertised in the catalog’s function listing, so DuckDB never registers it as a callable table function. Use this for the scan function behind a function-backed farm.query.vgi.catalog.CatalogTable that should surface only as a table (mirrors vgi-python, where a Table(function=F) does not imply F is in the catalog’s functions list).

java.util.Set<String> unlistedTables()

Names registered via #registerUnlistedTable: dispatchable, but omitted from catalog_schema_contents_functions.

Worker registerAggregates(Iterable<? extends AggregateFunction<?>> fns)

Register several aggregate functions; equivalent to calling #registerAggregate(AggregateFunction) for each.

Worker registerTableBuffering(farm.query.vgi.buffering.TableBufferingFunction fn)

Register a table-buffering (Sink+Source) function: DuckDB sinks the full input through table_buffering_process/_combine before the finalize stream sources results back out.

Worker registerTableBufferings(Iterable<? extends farm.query.vgi.buffering.TableBufferingFunction> fns)

Register several table-buffering functions; equivalent to calling #registerTableBuffering for each.

List<farm.query.vgi.buffering.TableBufferingFunction> bufferingFunctions()

Table-buffering functions registered via #registerTableBuffering.

Worker registerTableInOuts(Iterable<? extends TableInOutFunction> fns)

Register several table-in-out functions; equivalent to calling #registerTableInOut(TableInOutFunction) for each.

Worker registerGlobalFunctions(Iterable<? extends farm.query.vgi.function.FunctionDescriptor> fns)

Ask the client to publish these already-registered functions into its global (non-catalog) function namespace, under #globalFunctionPrefix(String). They are advertised on the catalog_attach result’s global_functions field as serialized FunctionInfo records (protocol 1.3.0).

Each argument must be the same instance passed to a register* method: the advertised FunctionInfo carries the schema the function is homed in, which is the bind-dispatch key, and instance identity is what #schemaOf(Object) resolves. Registration into the catalog is unchanged — publication is additive.

List<farm.query.vgi.function.FunctionDescriptor> globalFunctions()

Functions advertised via #registerGlobalFunctions.

Worker globalFunctionPrefix(String prefix)

Prefix the client applies to every #registerGlobalFunctions entry to form its globally visible name — e.g. "vgi_example" publishes global_scalar as vgi_example_global_scalar. An empty prefix publishes bare names.

String globalFunctionPrefix()

The prefix set by #globalFunctionPrefix(String).

Worker settings(SettingSpec… specs)

Advertise custom session settings in the catalog_attach result. DuckDB registers each as a SET-able option whose current value is forwarded to the worker on every bind.

Worker secretTypes(SecretTypeSpec… specs)

Advertise secret types in the catalog_attach result. DuckDB registers each so CREATE SECRET of that type resolves against this catalog, and matching secrets flow to the worker on bind.

List<SecretTypeSpec> secretTypeSpecs()

Secret types advertised at attach time.

Worker attachCatalogs(farm.query.vgi.protocol.AttachCatalogInfo… catalogs)

Advertise companion catalogs (lakehouse federation) that the client should ATTACH when this VGI catalog attaches. Surfaced via catalog_attach.attach_catalogs; the C++ extension attaches each at VGI-attach time so multi-branch catalog-table branches can resolve them.

List<farm.query.vgi.protocol.AttachCatalogInfo> attachCatalogInfos()

Companion catalogs advertised at attach time.

Worker attachOptions(AttachOptionSpec… specs)

Declare the options this worker accepts in DuckDB’s ATTACH ... (key value, ...) clause. Unknown options are rejected client-side; accepted values arrive in the attach request.

record SecretTypeSpec

source

public record SecretTypeSpec(String name, String description, Schema parametersSchema)

Description

Declares a DuckDB secret type backed by this worker. Mirrors vgi-go vgi.SecretTypeSpec.

Secret types are advertised at attach time via CatalogAttachResult.secret_types. Mark sensitive fields in parametersSchema with custom field metadata "redact":"true" so DuckDB can mask them in duckdb_secrets().

class Secrets

source

public final class Secrets

Description

Resolved secrets passed to a worker, keyed by each secret’s unique DuckDB secret name (not by type) so several secrets of the same type (e.g. one per S3 bucket) coexist. Each secret carries its connector-serialized type (the DuckDB secret type) and scope (newline-joined scope prefixes) fields, plus type-specific fields like key_id.

Mirrors vgi::Secrets in the Rust SDK. Parse the byte[] blob carried on the params with #parse(byte[]), then select by name, type, or scope.

Members

Secrets of(Map<String, Map<String, String>> byName)

Build directly from a name -> fields map (for tests / non-IPC callers).

Secrets parse(byte[] bytes)

Parse the IPC secrets blob. Each column is a secret (named by its DuckDB secret name) holding a struct of its fields, including type and scope. Empty/null blob yields empty secrets.

Optional<String> field(String field)

A field value from the first secret carrying it (any name).

Optional<String> namedField(String name, String field)

A named secret’s field.

Map<String, Map<String, String>> byName()

Every resolved secret as (name -> fields).

Optional<String> secretType(String name)

The DuckDB secret type of the named secret (its type field).

List<Map<String, String>> ofType(String secretType)

Every resolved secret whose type field matches secretType.

Optional<Map<String, String>> forScope(String path)

The fields of the secret whose scope is the longest prefix of path. The connector serializes each secret’s scope as a newline-joined list of prefixes; a secret with no (or empty) scope matches as a last-resort fallback. Empty only when there are no candidate secrets.

Optional<Map<String, String>> forScopeOfType(String path, String secretType)

Like #forScope but only over secrets of secretType.

Optional<String> fieldFor(String path, String field)

A field of the best scope-matching secret for path.

record SettingSpec

source

public record SettingSpec( String name, String description, ArrowType type, List<Field> children, Object defaultValue)

Description

Declares a custom DuckDB setting backed by this worker. Mirrors vgi-go vgi.SettingSpec.

Settings are advertised at attach time via CatalogAttachResult.settings and delivered with each bind call so functions can read them.

Members

SettingSpec(String name, String description, ArrowType type)

Scalar setting with no default.

SettingSpec(String name, String description, ArrowType type, Object defaultValue)

Scalar setting with a default value.

SettingSpec(String name, String description, ArrowType type, List<Field> children)

Complex (list/struct) setting with no default.

record TcpAddr

source

public record TcpAddr(String host, int port)

Description

Parsed [HOST:]PORT TCP bind spec. Host defaults to loopback.

Members

TcpAddr parseTcpAddr(String spec)

Parse a [HOST:]PORT TCP bind spec as accepted by --tcp. A bare PORT binds 127.0.0.1; an empty host (leading ":") also defaults to loopback.

void runHttp(String host, int port) throws Exception

Run as an HTTP server bound to host/port, blocking until shutdown.

void runFromArgs(String[] args, java.util.function.UnaryOperator<HttpServer.Config.Builder> httpCustomizer)

Canonical CLI dispatcher used by worker main methods. Parses the four flags every VGI worker accepts and runs the matching transport:

• --unix <path>: AF_UNIX socket (launcher protocol)

• --tcp [<host>:]<port>: TCP socket (launcher protocol)

• --http with optional --host, --port: HTTP

• --idle-timeout <seconds>: passed to runUnixSocket / runTcp

• (default): stdio

Also honours VGI_WORKER_STDERR: redirects System#err to the named file (in append mode) before any other work, so launcher-mode crashes — where the launcher dup2’s /dev/null over fd 2 — remain inspectable.

Unknown args exit with status 2; transport-run failures with 1.

void runFromArgs(String[] args)

Convenience overload — equivalent to runFromArgs(args, b -> b).

void runHttp(HttpServer.Config config) throws Exception

HTTP variant that accepts a fully-built config (prefix, authenticator, TLS, byte limits, …). Used by workers that wire OAuth/JWT or other production knobs from environment variables.

On SIGTERM the shutdown hook fires, calling HttpServer#stop(). Jetty awaits in-flight requests up to its configured stop timeout (15 s by default; see HttpServer’s setStopTimeout) and then forcibly closes any stragglers.

interface VgiService

source

public interface VgiService

Description

The VGI RPC surface served by a worker.

Wire shape varies per method. Two flavours coexist:

• Packed — the params batch has a single request: binary column carrying a serialised farm.query.vgirpc.schema.ArrowSerializableRecord payload. Used by bind/init/catalog_attach and any method whose request would contain maps/lists/structs.

• Flat — the params batch’s columns map 1:1 to the method’s parameters by snake_case name. Used by simpler catalog reads/writes.

Method and parameter / record-field names are the wire contract — they MUST match the canonical Python/Go snake_case.

The ProtocolVersion is what a client of this interface stamps on every request. A VGI worker enforces it at its dispatch boundary (exact major+minor), so a client that sends nothing is refused outright — the annotation is what lets connection.proxy(VgiService.class) talk to vgi-python or vgi-go. It shares Worker#VGI_PROTOCOL_VERSION with the server side so the two cannot drift.

class Worker

source

public final class Worker

Description

Builder + run-loop façade for a VGI worker.

Mirrors vgi.Worker in vgi-go: register functions, configure catalog metadata, then call #runStdio() or #runHttp(String, int).

Members

String VGI_PROTOCOL_VERSION = “1.3.0”

VGI protocol surface version. Mirrors vgi-python protocol_version.txt. Emitted as the vgi_rpc.protocol_version per-request metadata key.

1.1.0 added the nullable schema_name field to the bind request: a function name is not a unique key, because the same name may be registered in more than one catalog schema, so dispatch resolves (schema_name, function_name).

1.3.0 added global_functions / global_function_prefix to the catalog_attach result (positions 14/15, before resolved_data_version): functions a worker asks the client to publish into its global namespace — see #registerGlobalFunctions and #globalFunctionPrefix(String). A worker that opts out still carries the fields (empty list, empty prefix): the extension matches the response schema exactly.

Worker registerMacro(Macro m)

Register a SQL macro.

Worker registerMacros(Iterable<? extends Macro> ms)

Register several SQL macros.

List<Macro> macros()

Macros enumerated through catalog_schema_contents_macros.

Worker registerCatalogTable(CatalogTable t)

Register a catalog table.

List<CatalogTable> catalogTables()

Catalog tables enumerated through catalog_schema_contents_tables / catalog_table_get.

Worker registerMultiBranchTable(CatalogTable stub, List<farm.query.vgi.catalog.ScanBranch> branches)

Register a multi-branch table: a catalog table whose scan is the UNION_ALL of branches. The table is enumerated normally; its scan resolves through catalog_table_scan_branches_get. Pass an empty branch list to exercise the C++ loud-fail path. The stub’s inline scan-function (if any) is dropped so the branches RPC drives.

Worker registerMultiBranchTable(CatalogTable stub, List<farm.query.vgi.catalog.ScanBranch> branches, List<String> requiredExtensions)

Register a multi-branch table declaring the DuckDB extensions the C++ rewriter must auto-load before binding any branch (e.g. "iceberg" for an iceberg_scan arm, "parquet" for read_parquet where it isn’t autoloaded). Surfaced as the required_extensions field of the catalog_table_scan_branches_get response.

List<farm.query.vgi.catalog.ScanBranch> multiBranchTable(String schema, String name)

Branches for a multi-branch table.

List<String> multiBranchRequiredExtensions(String schema, String name)

DuckDB extensions the C++ rewriter must auto-load for a multi-branch table’s branches, or an empty list when none were declared.

Map<String, List<farm.query.vgi.catalog.ScanBranch>> multiBranchTables()

Every table registered via #registerMultiBranchTable(CatalogTable, List), used by the service to answer catalog_table_scan_branches_get.

Worker builder()

Start building a worker. Defaults: catalog name "vgi", default schema "main", empty comment/tags, no versioning metadata.

Worker catalogName(String name)

Name this worker’s catalog. Surfaced as the catalog row in catalog_catalogs() and as the default database alias on ATTACH.

Worker catalogComment(String comment)

Set the catalog-level comment, surfaced through catalog_catalogs() and DuckDB’s duckdb_databases() comment column.

Worker catalogTags(Map<String, String> tags)

Attach key/value tags to the catalog, surfaced through catalog_catalogs(). Merged into any previously set tags (later calls overwrite duplicate keys).

Worker implementationVersion(String v)

Advertise the worker’s implementation (code) version, reported through catalog_version alongside the resolved data version so clients can distinguish “what code is running” from “what data it serves”.

Worker dataVersionSpec(String v)

Declare the range of data versions this worker can serve. ATTACH-time version requests are validated against this spec; requests outside the range are rejected.

String implementationVersion()

Implementation version advertised through catalog_version.

String dataVersionSpec()

Data-version range this worker accepts at ATTACH time.

Worker opaqueDataKey(byte[] key)

Provide a stable 32-byte key for sealing attach / transaction opaque_data. Required when running the same worker across multiple HTTP replicas: without it each replica generates its own random key, and a load balancer rotating across them will surface AEADBadTagException when one replica receives a blob another replica sealed.

null (the default) restores the per-process random-key behaviour, which is correct for single-replica HTTP and irrelevant for stdio / AF_UNIX (where the sealer is disabled entirely).

Worker releases(CatalogDataVersionRelease… rs)

Published data-version releases, surfaced through catalog_catalogs(). Pass newest-first.

List<CatalogDataVersionRelease> releases()

Data-version releases surfaced through catalog_catalogs().

Worker sourceUrl(String url)

Set the catalog’s source URL (e.g. a homepage or repository link), surfaced through catalog_catalogs().

String sourceUrl()

Source URL surfaced through catalog_catalogs().

Worker defaultSchema(String schema)

Name the schema DuckDB selects by default after ATTACH. Functions, tables and views without an explicit schema register here.

Worker schemaComment(String schema, String comment)

Per-schema comment surfaced via catalog_schemas / catalog_schema_get. Default comment for the default schema is “Default schema”; any auxiliary schema without an entry gets an empty comment.

Map<String, String> schemaComments()

Comments registered via #schemaComment(String, String).

Worker schemaTags(String schema, Map<String, String> tags)

Attach key/value metadata tags to a schema, surfaced via catalog_schemas / catalog_schema_get and reported through DuckDB’s duckdb_schemas().tags. Typical keys are vgi.description_llm and vgi.description_md. Merged into any tags previously set for the schema (later calls overwrite duplicate keys).

Map<String, Map<String, String>> schemaTags()

Tags registered via #schemaTags(String, Map).

class WorkerLandingInfo

source

public final class WorkerLandingInfo

Description

Derives a LandingInfo from a Worker.

The shared landing page reads catalog metadata over the VGI protocol through the client bundle the transport serves beside it, so the worker supplies only what the protocol has no method for: which worker this is, what it is called, and what version it runs.

Members

LandingInfo of(Worker worker)

Build the landing identity for worker: name from the catalog name, doc from the catalog comment’s first line, version from the VGI core package manifest.