Skip to content
Query.Farm
Talk with Us

Settings, secrets, and global functions

Four smaller capabilities a worker declares on itself rather than on a function: settings, secrets, global functions, and transaction support. All are WorkerOptions passed to vgi.NewWorker.

Settings

Section titled “Settings”

A worker can register its own DuckDB settings — the user sets them with SET, and their values reach your functions on params.Settings.

w := vgi.NewWorker(
  vgi.WithCatalogName("acme"),
  vgi.WithSettings(
      vgi.SettingSpec{
          Name:         "acme_timeout_ms",
          Description:  "How long to wait on the upstream before giving up",
          Type:         arrow.PrimitiveTypes.Int64,
          DefaultValue: int64(5000),
      },
  ),
)
SET acme_timeout_ms = 250;
SELECT * FROM acme.data.slow_thing();

Read the value inside a function from params.Settings, keyed by name. There is no way to declare that a function requires a setting — FunctionInfo has a RequiredSettings field but nothing populates it from metadata — so validate it yourself at bind and return a clear error if it’s missing.

Secrets

Section titled “Secrets”

Secrets are DuckDB’s credential mechanism. Declaring a type tells DuckDB what fields your secret has; users then create instances with CREATE SECRET, and the resolved values arrive on params.Secrets.

vgi.WithSecretTypes(
  vgi.SecretTypeSpec{
      Name:        "acme",
      Description: "Credentials for the Acme API",
      Schema: arrow.NewSchema([]arrow.Field{
          {
              Name: "api_key", Type: arrow.BinaryTypes.String,
              Metadata: arrow.NewMetadata([]string{"redact"}, []string{"true"}),
          },
          {Name: "endpoint", Type: arrow.BinaryTypes.String},
      }, nil),
  },
)
CREATE SECRET acme_prod (TYPE acme, api_key 'sk-...', endpoint 'https://api.acme.test');
Mark sensitive fields redact

The {"redact": "true"} field metadata is what keeps a value out of duckdb_secrets() and out of error messages. It is per-field and opt-in, so a field you forget to mark is a field that shows up in plain text — mark every credential, and leave only genuinely non-sensitive fields (an endpoint, a region, a port) unmarked.

Inside a function, params.Secrets is a map[string]map[string]interface{} keyed by secret name, then field. vgi.RenderSecretValue turns a field into a string regardless of its Arrow type.

Global functions

Section titled “Global functions”

A catalog normally exposes its functions under its own name — acme.main.thing(). Some functions aren’t about the catalog’s data at all: diagnostics, converters, format helpers. Those can be published into DuckDB’s global namespace so they’re callable unqualified.

w := vgi.NewWorker(
  vgi.WithCatalogName("acme"),
  vgi.WithGlobalFunctions("table_info", "checksum"),
  vgi.WithGlobalFunctionPrefix("acme"),   // published as acme_table_info, acme_checksum
)
ATTACH 'acme' (TYPE vgi, LOCATION './acmeworker');
SELECT * FROM acme_table_info('acme');   -- no catalog qualifier

Names are resolved against the catalog’s default schema, and advertising one does not move it: the function stays an ordinary member of its schema and remains callable as acme.main.table_info().

Registration is best-effort — prefer a prefix

system.main is shared by every extension and every attached worker in the process. If the name is already taken, yours is skipped and logged; your ATTACH still succeeds and the qualified name still works. Treat a global name as an ergonomic alias, never as something to depend on — and set a prefix, because an unprefixed table_info is very likely to be claimed by someone else.

Registration also outlives DETACH: DuckDB has no API to unregister a function, so the entry persists for the life of the process.

The full semantics — first attach wins, re-attach is idempotent, vgi_global_functions() for introspection — are the same across SDKs and documented once on Publish global functions.

Transactions

Section titled “Transactions”

If your catalog needs to see DuckDB’s transaction boundaries — because it caches per transaction, or talks to a source with its own transactions — opt in:

vgi.WithSupportsTransactions(true)

DuckDB then threads a transaction_opaque_data through bind and scan inside BEGIN/COMMIT, so a function can tell one transaction from another. Without it those methods are never called, which is the right default for a stateless worker.

Transaction-scoped caching

CacheScopeTransaction on a CacheControl is the other half of this: a result marked with it is reused only within the transaction that produced it. See Cache results on the client.

Next steps

Section titled “Next steps”