Concepts

The mental model behind VGI — what it’s for, and how the pieces fit. This page is about the protocol, so it applies whichever SDK you use. Start here if you want the “why” before the “how”; to just build something, do a tutorial: Python · Go · TypeScript · Rust · Java.

Where the engineering case lives

This page explains what VGI is. The Architecture page argues the why — what a native extension actually costs you, what the process boundary costs in throughput, how a worker gets hosted, and where VGI sits against Airport and ADBC. It also puts upper_case(VARCHAR) side by side in DuckDB’s C API and in all five SDKs, which is the fastest way to see what the SDK is doing for you. Sections below link into it rather than repeat it.

The problem VGI solves

DuckDB is a fast SQL database that runs inside your application. It’s great out of the box, but eventually you want it to do something it can’t — call your ML model, hit an internal API, run a custom transform — from SQL.

New to DuckDB?

DuckDB is an in-process analytical SQL database (think “SQLite for analytics”): no server to run, it executes queries right in your program. You query it with ordinary SQL.

The usual way to add a new capability is a DuckDB extension. Classic extensions are C++ compiled against DuckDB’s internals; more recently DuckDB has stabilised a C extension API that decouples from those internals and opens the door to other languages (Rust, Go, …). Either way, though, an extension still runs in-process — a crash can take the whole database down — and, more practically, you stay tied to DuckDB’s release and distribution cadence: you build against specific DuckDB versions and distribute through its community-extension repository, so shipping a new version of your extension depends on the DuckDB team’s latency in approving and merging your release.

What's a DuckDB extension?

An extension is compiled code (a shared library) DuckDB loads to add new SQL functions, types, or file formats — like httpfs or spatial. It runs inside the DuckDB process — traditionally in C++, though DuckDB’s stable C extension API now allows other languages.

Your function is compiled into the engine and shipped per DuckDB release — fast, but in-process (a crash can take the database down) and on DuckDB’s release cadence:

VGI takes a different approach. Instead of compiling code into DuckDB, you write your function as an ordinary separate program — a worker — in whatever language you like. DuckDB launches it and talks to it out-of-process. From SQL, your function looks completely native — but it’s an independent program you build, ship, and update on your own schedule, not DuckDB’s.

What does “out-of-process” mean?

Your code runs in its own operating-system process, not inside DuckDB’s. The two exchange data over a connection (a pipe, socket, or HTTP). The upside: no compiling against DuckDB, use any library, a crash in your worker can’t take down the database, and you ship on your own release cadence.

Now your function is a separate process the engine talks to over a transport using Apache Arrow:

In short: VGI lets you add functions, tables, and catalogs to DuckDB from an external program, in any language. No native extension to build or ship.

The big picture

DuckDB is the client; your worker is the server. DuckDB launches your worker (the LOCATION in ATTACH) and they exchange columns of data as Apache Arrow record batches.

New to Apache Arrow?

Apache Arrow is a columnar in-memory format: data moves as typed arrays (columns) rather than rows of objects, which is what keeps it fast across the process boundary. Your function receives whole columns and operates on them at once, through whatever Arrow library your language uses — PyArrow in Python, arrow-go in Go. The Python tutorial’s primer has a fuller introduction to the format.

Two things are named “vgi”

Keep them straight: your SDK (vgi-python, vgi-go, @query-farm/vgi) is the package you write your worker with, and the vgi extension is the engine-side piece (the TYPE vgi in ATTACH) that lets DuckDB launch and talk to it. One extension serves every language. It ships with Haybarn; to drive it from your own program rather than the CLI, see Use VGI from a Python app.

Isn’t leaving the process slow?

It’s the first thing everyone asks, and it’s worth answering before you read any further. Four things make the boundary far cheaper than it sounds:

• The unit is a batch, not a row. DuckDB is vectorized — it hands over thousands of rows per call. A round trip is amortized across the whole vector, so the hop is a fixed cost per batch, not a tax on every row.
• Arrow on both sides means no conversion. DuckDB’s execution format and your worker’s are the same columnar layout, so there is no serialize/deserialize step in the middle. On one host, shared memory moves the buffers with no copy at all.
• Fewer calls, not just cheaper ones. Projection, filter, ORDER BY + LIMIT and distinct join-key pushdown mean the worker sends back less; catalogs load lazily; subprocess workers are pooled and reused rather than spawned per query.
• It measures at roughly 450 million rows/second across the boundary — fast enough that for most workloads the hop stops being the thing you tune.

The honest exception: a scalar function on a genuinely hot path — called a billion-plus times a second, doing almost nothing per call — is faster in-process, and batching doesn’t close that gap. The Architecture page unpacks all of this, including when to reach for a native extension instead.

What you build

You expose functions — grouped into a catalog you ATTACH — in one of five shapes. Pick the one whose input/output cardinality matches your problem; each links to a complete, runnable example.

Scalar1 row → 1 valueTableargs → N rowsTable-in-outN rows → M rowsAggregateN rows → 1 valueBufferingstream → [state] → stream

Argument and result types are declared in your language’s own idiom — Python type annotations, Go struct tags — and VGI derives the SQL signature from them. The exact phases each shape runs through — bind, init, process, finalize — are diagrammed in the Function lifecycle reference.

A worker can be a whole database

Functions are the small case. A worker can also expose schemas, tables and views, so you ATTACH it and query it like any other database — and DuckDB plans against it with the same kind of information it has about a local table. Metadata and per-column statistics load lazily, on demand, and a worker can opt into projection, filter, ORDER BY + LIMIT and distinct join-key pushdown, so the planner tells it what the query actually needs and the answer comes back smaller than the table. See Workers are first-class catalogs, then Expose a catalog in your SDK.

Under the hood (optional)

You don’t need any of this to ship a worker — but if you’re curious how the calls actually travel:

VGI is a protocol, not just a library: a fixed set of operations (bind a function, process batches, describe a catalog) that the engine and worker agree on. Those operations are carried by vgi-rpc, a transport-agnostic RPC framework that serializes every request and response as Apache Arrow IPC. VGI defines what the calls are; vgi-rpc defines how they travel — so the same worker runs unchanged over any transport, and any language with an Arrow library can implement either side.

• Transports — the same worker runs over subprocess (the default; stdin/stdout), a Unix domain socket, TCP, or HTTP. For co-located processes vgi-rpc can also use shared memory for zero-copy transfer. Only HTTP authenticates callers — subprocess and Unix sockets are co-located and trusted, and raw TCP carries no auth or encryption, so it belongs on loopback or a trusted network. Serve over HTTP: Python · Go · TypeScript, with optional authentication.
• Where the worker runs — because HTTP is just HTTP, a worker deploys onto ordinary web infrastructure: spawned locally beside DuckDB, run as a long-lived service, or dropped into a serverless runtime behind a URL. The trade-offs are laid out in Where a worker actually runs.
• The full byte-level contract is the vgi-rpc wire protocol specification.

Inspired by DuckDB's own APIs

The VGI surface deliberately mirrors DuckDB’s internal and extension APIs: scalar, table, table-in-out, and aggregate functions; the bind → init → execute → finalize phasing; and catalogs with schemas, tables, and views. If you’ve written a DuckDB extension, the shapes will feel familiar — VGI exposes the same concepts, out-of-process and in the language of your choice. To see exactly what that costs and saves, the Architecture page puts upper_case(VARCHAR) in DuckDB’s C API next to the same function in all five SDKs.

When VGI isn’t the right tool

Worth knowing up front, so you don’t find out three days in:

• A genuinely hot scalar path — billion-plus calls a second doing almost nothing per call — is faster as a native in-process extension, and no amount of batching closes that gap.
• Querying an existing database (Postgres, Snowflake, …) is a client’s job, not a worker’s — use ADBC Scanner. VGI is for making your code part of DuckDB.
• Arrow Flight is already your wire — Airport speaks it natively and stays the right call. VGI came out of Airport and swapped Flight’s gRPC/HTTP-2 constraint for plain Arrow IPC over a transport you choose.
• You can’t run or host a worker at all — then there is nothing for DuckDB to talk to.

The full comparison is VGI vs. native extensions, Airport & ADBC.

Next steps

• The engineering case → Architecture — the cost of a native extension, the latency question, hosting, and the side-by-side with DuckDB’s C API.
• Build one → the tutorial: Python · Go · TypeScript · Rust · Java.
• Pick a shape → Function patterns: Python · Go · TypeScript · Rust · Java.
• The phases in detail → Function lifecycle.
• How types cross the wire → Argument serialization.
• The RPC layer underneath → vgi-rpc and its wire protocol.