Serve over HTTP

Run a worker as a network service instead of a co-located subprocess — on another machine, in a container, or behind a shared endpoint.

Prerequisites

A working worker (see the tutorial).

The four transports

The same worker serves all of them; only the Run* call at the bottom of main changes.

Call	Transport	Readiness marker	Authenticates?
`RunStdio()`	stdin/stdout — the default, and what `ATTACH … LOCATION ‘./worker’` uses	—	Co-located, trusted
`RunUnix(path, idle)`	AF_UNIX socket	`UNIX:<path>`	Co-located, trusted
`RunTcp(host, port, idle)`	Raw Arrow IPC over TCP	`TCP:<host>:<port>`	No — see below
`RunHttp(addr)`	HTTP	`PORT:<n>`	Only if you configure it

The example workers in these docs all take an --http flag for exactly this reason:

func main() {
  httpMode := flag.Bool("http", false, "serve over HTTP instead of stdio")
  flag.Parse()

  w := vgi.NewWorker(vgi.WithCatalogName("calc"))
  w.RegisterScalar(NewDouble())

  if *httpMode {
      if err := w.RunHttp("127.0.0.1:0"); err != nil {
          log.Fatal(err)
      }
      return
  }
  w.RunStdio()
}

RunHttp("127.0.0.1:0") binds a free port and prints PORT:<n> to stdout so a supervisor can discover it. Pin the port by passing it explicitly.

Attach it from DuckDB

DuckDB attaches an HTTP worker the same way as a subprocess one — point LOCATION at the URL:

ATTACH 'calc' (TYPE vgi, LOCATION 'http://localhost:8080');
SELECT calc.double(21);

Set a signing key before running more than one process

The HTTP transport seals two things with an HMAC key: the state tokens that carry a scan’s cursor between requests, and the catalog opaque data returned by ATTACH. WithHttpSigningKey sets it.

If you don’t, RunHttp mints an ephemeral key per process.

An ephemeral key is correct for one process and silently wrong for more

A client whose connection stays pinned to one process never notices. One that reconnects mid-stream — a load balancer, a respawned worker, a resumed scan — presents a token sealed under a key the receiving process does not have, and gets an intermittent failure that reads as flakiness rather than as misconfiguration.

Set the key explicitly for any load-balanced or multi-instance deployment. Nothing in the process can reach a peer it did not start, so it cannot detect the problem for you.

key, err := hex.DecodeString(os.Getenv("VGI_SIGNING_KEY"))
if err != nil || len(key) == 0 {
  log.Fatal("VGI_SIGNING_KEY must be set for a multi-process deployment")
}

w := vgi.NewWorker(
  vgi.WithCatalogName("calc"),
  vgi.WithHttpSigningKey(key),
)

A restart with an ephemeral key invalidates every sealed value, so clients must re-ATTACH. With a configured key they survive it.

A bare RunHttp is not authenticated, and not encrypted

HTTP is the only transport that can authenticate, but it does not by default: auth is wired only when you call w.SetAuthenticate(...), and without it every request is anonymous. RunHttp also serves plain HTTP — there is no TLS unless you put a terminating proxy in front. So an unconfigured HTTP worker is exactly as open as the TCP one below; the difference is that HTTP gives you somewhere to attach identity. See Authenticate callers.

TCP, for trusted networks

RunTcp serves raw Arrow IPC with no HTTP framing in the path — lower overhead, and useful for a co-located sidecar.

if err := w.RunTcp("127.0.0.1", 9000, 5*time.Minute); err != nil {
  log.Fatal(err)
}

TCP carries no auth and no encryption

Anyone who can reach the port can call every function. Bind loopback or a trusted network, and use HTTP the moment the network isn’t one. This is the same trade the Python SDK makes — it’s a property of the transport, not of the language.

host: "" defaults to 127.0.0.1 and port: 0 picks a free one. The idleTimeout shuts the worker down after that long with no active connections; pass <= 0 to disable it.

Logging

Every example worker wires up the SDK’s logging flags, which is what makes a deployed worker debuggable without a rebuild:

logFlags := vgi.RegisterLoggingFlags(flag.CommandLine)
flag.Parse()
if err := logFlags.Apply(); err != nil {
  log.Fatalf("logging flags: %v", err)
}

Logs go to stderr — they must, because on the stdio transport stdout is the protocol. The flags are --log-level, --log-format (text or json), --log-logger, and --debug.

One level applies to every logger; --log-logger selects which of the named loggers (vgi.Log, vgi.LogWorker, vgi.LogCatalog, vgi.LogRPC, …) emit at all, rather than giving each its own level. See Errors & logging.

Next steps

Lock it down → Authenticate callers.
What a worker exposes → Expose a catalog.
The transport underneath → vgi-rpc.
Exact signatures → Worker & serving.