Skip to content
Query.Farm
Talk with Us

Use VGI from a Python app

Run a worker from your own Python program — when your app already talks to DuckDB and you want VGI functions in the same process, without the Haybarn CLI.

Prerequisites

Section titled “Prerequisites”
  • A worker you can run (see the tutorial).
  • pip install haybarn vgi-python — haybarn is the in-process engine (drop-in compatible with the duckdb Python API) that ships the vgi extension; vgi-python is the worker library.

Connect and attach

Section titled “Connect and attach”

haybarn.connect() returns the same connection object as duckdb.connect(), so the rest is ordinary DuckDB SQL — load vgi, attach your worker, and query:

import haybarn

con = haybarn.connect()                       # in-memory; same API as duckdb.connect()
con.execute("INSTALL vgi FROM community; LOAD vgi;")
con.execute("ATTACH 'calc' (TYPE vgi, LOCATION 'uv run calc_worker.py')")

con.execute("SELECT calc.double(21)").fetchone()        # (42,)
con.execute("SELECT * FROM calc.series(3)").fetchall()  # [(0,), (1,), (2,)]
Why haybarn, not the duckdb package?

The vgi extension is distributed through Haybarn’s channel, not DuckDB’s public community repository — so a plain import duckdb can’t INSTALL vgi today. haybarn is DuckDB-compatible (import haybarn as duckdb works) and carries the extension, so use it as your engine.

Calling a worker without an engine

Section titled “Calling a worker without an engine”

To invoke a worker’s functions directly from Python — for tests or scripts, with no DuckDB engine in the loop — use the Client instead. It speaks to the worker over the subprocess, HTTP or TCP transport and exposes the same calls. See Serve over HTTP for the HTTP variant.

Every call names its schema

Section titled “Every call names its schema”
Breaking change in 0.18.0

All five Client entry points now require schema_name, and vgi-client gained --schema. Code written against 0.17 and earlier will raise a TypeError until you add it.

A function name was never a unique key: a worker may register the same name in several catalog schemas, and dispatch used to be a flat name lookup, so two such functions collided as overloads and the call failed as “Ambiguous function call”. Resolution is now keyed on (schema, name), which makes a schema-qualified lookup exact:

from vgi.client import Client

with Client(server_path="uv run calc_worker.py") as client:
  for batch in client.table_function(
      function_name="series",
      schema_name="main",       # required since 0.18.0
      arguments=Arguments(...),
  ):
      ...

Functions declared in the legacy flat functions list register into the catalog’s default_schema, which is the schema DuckDB registers them into — so schema_name="main" is usually what you want.

Spawning a worker

Section titled “Spawning a worker”

server_path accepts a sequence as well as a string (since 0.28.1). Take the sequence form whenever an argument contains spaces or quotes: splitting a command string is ambiguous on Windows, where shlex must run with posix=False so backslashes in paths survive, which in turn leaves quotes attached to the token.

Client(server_path=["python", "-c", "import sys; run_worker(sys.argv)"])

Workers are spawned without a shell, so the client owns the real process rather than a shell wrapping it — which is what makes the next section work.

Cancelling a call that has overrun

Section titled “Cancelling a call that has overrun”

A graceful stop() closes the stream and shuts the RPC connection down first, and both block on a worker wedged inside a handler — exactly the case you want to escape. stop(force=True) SIGKILLs direct subprocess workers first, unblocking any thread waiting on their output:

client.stop(force=True)   # returns the signal exit code (-9 on POSIX)

This only applies to direct subprocess workers. A pooled worker is returned to its pool rather than owned by the client, so force is inert there — construct the client with pool=None when you need to be able to cancel it. HTTP and TCP workers are already prompt to close.

Reading column statistics

Section titled “Reading column statistics”

Client.table_column_statistics() fetches and decodes a table’s statistics in one call, so a non-DuckDB client no longer has to re-implement the sparse-union layout to read the bytes from catalog_table_column_statistics_get. The standalone inverse is deserialize_column_statistics(). See Column statistics.

Next steps

Section titled “Next steps”