DuckDB extensions, without the DuckDB internals.

The program on the far end is called a worker, and it is ordinary code — no DuckDB headers, no build against its internals. This page works top down: why that's worth doing, the five shapes a worker's functions can take, how it's reached and hosted, and finally the wire itself.

Where this came from

Extensions are how DuckDB grows — and the hardest thing to ship

Parquet, cloud storage, geospatial, half a dozen other databases: nearly everything DuckDB can reach arrives as an extension, and the interface for writing one is genuinely good. What's hard is everything around writing it. Two costs in particular land before your code ever answers a query.

Neither is a complaint about DuckDB. Both are the honest price of running inside the engine's process, and if you're building one extension, it's a price worth paying. We wanted to build them at scale — dozens of them, maintained by people who shouldn't have to learn DuckDB's internals first, and increasingly written with the help of coding agents. Paid once, that price is fine. Paid per extension, it doesn't divide.

The idea

Let the extension live outside DuckDB

So we wrote one extension, generically, and gave it nothing to do but talk. It's an ordinary DuckDB extension — compiled, signed, distributed, all of it — and its only job is to carry DuckDB's requests to a program running outside the database, then bring the answers back. That program is a worker: your code, in whatever language you like, answering in Apache Arrow.

From the SQL side, nothing looks unusual. You ATTACH a worker, DuckDB sees catalogs, schemas, tables, and functions — its own objects, with its own semantics — and it plans queries against them the way it plans anything else.

That one move takes care of both costs, and a few other things along with them:

The obvious objection

“Isn't leaving the process slow?”

It's the first question everyone asks, and it deserves a real answer rather than a benchmark screenshot. Four things make that boundary far cheaper than it sounds. Then there's the one case where the worry is justified.

And the case where it's justified: a scalar function on a genuinely hot path — called a billion-plus times a second, doing almost nothing per call — runs faster in-process, and no amount of batching closes that gap. A chatty pattern that can't be expressed set-at-a-time pays the boundary over and over for the same reason. For those, a native extension is still the right tool, and we'd rather say so than sell you around it.

Lineage

Airport, retooled for the web

VGI is not the first attempt at this. We built Airport first — a DuckDB extension that reaches remote services over Arrow Flight and exposes them as catalogs, tables, and functions. Airport proved the idea works, and it is still the right answer if you already speak Flight.

What it inherited from Flight was the constraint. Flight is gRPC and Protocol Buffers: excellent inside a data center, awkward everywhere else. Requiring end-to-end HTTP/2 with trailer support rules out a lot of ordinary web infrastructure — CDNs, proxies, edge and function-as-a-service runtimes — and Flight's verbs describe a narrower slice of DuckDB than an extension can actually register.

VGI is that idea retooled. Same premise, different wire: Apache Arrow IPC over a transport you choose, with plain HTTP as the web-friendly default. That bought back the entire hosting story below, along with a wider surface — five function shapes, per-column statistics, pushdown, and transactional catalog semantics. It's faster, too.

The interface

The same API, one hop out of the process

Here's the part that matters if you've written against duckdb.h: VGI deliberately mirrors DuckDB's own extension interfaces rather than inventing a friendlier abstraction on top of them. If you know what bind does, or why an aggregate needs combine, you already know how a worker is structured — the concepts are DuckDB's, and only the plumbing underneath them changed.

Take upper_case(VARCHAR) → VARCHAR, about as small as a function gets. The first tab is what it takes to write natively, in C; the rest are the same function as a VGI worker, one per SDK. Each one is a whole program, not an excerpt, and each registers upper_case in a catalog DuckDB can ATTACH.

/* DuckDB C extension API — upper_case(VARCHAR) -> VARCHAR: the
   compute callback and the registration that publishes it. */
static void upper_case(duckdb_function_info info,
                       duckdb_data_chunk input,
                       duckdb_vector output) {
  idx_t count             = duckdb_data_chunk_get_size(input);
  duckdb_vector in        = duckdb_data_chunk_get_vector(input, 0);
  duckdb_string_t *in_data = (duckdb_string_t *) duckdb_vector_get_data(in);
  uint64_t *in_valid      = duckdb_vector_get_validity(in);

  duckdb_vector_ensure_validity_writable(output);
  uint64_t *out_valid = duckdb_vector_get_validity(output);

  for (idx_t row = 0; row < count; row++) {
    if (!duckdb_validity_row_is_valid(in_valid, row)) {
      duckdb_validity_set_row_invalid(out_valid, row);
      continue;
    }

    /* Strings up to 12 bytes live inside the struct; longer ones are behind a
       pointer — you have to know which before you can read one. */
    duckdb_string_t str = in_data[row];
    const char *bytes   = duckdb_string_is_inlined(str) ? str.value.inlined.inlined
                                                        : str.value.pointer.ptr;
    uint32_t len        = duckdb_string_is_inlined(str) ? str.value.inlined.length
                                                        : str.value.pointer.length;

    char *upper = (char *) malloc(len);   /* ASCII fast path only */
    for (uint32_t i = 0; i < len; i++) {
      upper[i] = (char) toupper((unsigned char) bytes[i]);
    }
    duckdb_vector_assign_string_element_len(output, row, upper, len);
    free(upper);
  }
}

DUCKDB_EXTENSION_ENTRYPOINT(duckdb_connection conn,
                            duckdb_extension_info info,
                            struct duckdb_extension_access *access) {
  duckdb_logical_type varchar = duckdb_create_logical_type(DUCKDB_TYPE_VARCHAR);
  duckdb_scalar_function fn   = duckdb_create_scalar_function();

  duckdb_scalar_function_set_name(fn, "upper_case");
  duckdb_scalar_function_add_parameter(fn, varchar);
  duckdb_scalar_function_set_return_type(fn, varchar);
  duckdb_scalar_function_set_function(fn, upper_case);
  duckdb_register_scalar_function(conn, fn);

  duckdb_destroy_scalar_function(&fn);
  duckdb_destroy_logical_type(&varchar);
  return true;
}

from typing import Annotated

import pyarrow as pa
import pyarrow.compute as pc

from vgi import Param, Returns, ScalarFunction, Worker
from vgi.catalog import Catalog, Schema

class UpperCase(ScalarFunction):
    """Converts string values to uppercase."""

    class Meta:
        name = "upper_case"
        description = "Converts string values to uppercase"

    @classmethod
    def compute(
        cls,
        value: Annotated[pa.StringArray, Param(doc="String value to uppercase")],
    ) -> Annotated[pa.StringArray, Returns()]:
        return pc.utf8_upper(value)

class DemoWorker(Worker):
    catalog = Catalog(
        name="demo",
        schemas=[Schema(name="main", functions=[UpperCase])],
    )

if __name__ == "__main__":
    DemoWorker().run()

import { Worker, defineScalarFunction, str } from "@query-farm/vgi";

const upperCase = defineScalarFunction({
  name: "upper_case",
  description: "Convert a string to uppercase",

  // Declares the argument types and order. DuckDB calls scalar functions
  // positionally, so inside compute you read columns by position.
  params: { value: str },
  returns: str,

  // compute runs once per input batch: one output value per row (null ok).
  compute: (batch) => {
    const values = batch.getChildAt(0)!;     // first positional argument
    return Array.from(values, (v) =>
      v == null ? null : String(v).toUpperCase(),
    );
  },
});

// Functions are served through a catalog — that's what DuckDB ATTACHes to.
new Worker({
  catalog: { name: "demo", schemas: [{ name: "main", functions: [upperCase] }] },
}).run();

package main

import (
	"context"
	"strings"

	"github.com/Query-farm/vgi-go/vgi"
	"github.com/apache/arrow-go/v18/arrow"
	"github.com/apache/arrow-go/v18/arrow/array"
)

// UpperCaseFunction converts string values to uppercase.
type UpperCaseFunction struct{}

func (f *UpperCaseFunction) Name() string { return "upper_case" }

func (f *UpperCaseFunction) Metadata() vgi.FunctionMetadata {
	return vgi.FunctionMetadata{
		Description: "Converts string values to uppercase",
		Stability:   vgi.StabilityConsistent,
		ReturnType:  arrow.BinaryTypes.String,
	}
}

func (f *UpperCaseFunction) ArgumentSpecs() []vgi.ArgSpec {
	return []vgi.ArgSpec{
		{Name: "value", Position: 0, ArrowType: "varchar", Doc: "String value to uppercase"},
	}
}

func (f *UpperCaseFunction) OnBind(params *vgi.BindParams) (*vgi.BindResponse, error) {
	return vgi.BindResult(arrow.BinaryTypes.String)
}

func (f *UpperCaseFunction) Process(ctx context.Context, params *vgi.ProcessParams, batch arrow.RecordBatch) (arrow.RecordBatch, error) {
	get := vgi.StringAccessor(batch.Column(0)) // hoist the type switch out of the row loop
	return vgi.MapColumn(params, batch, 0, array.NewStringBuilder,
		func(_ arrow.Array, i int) string {
			return strings.ToUpper(get(i))
		})
}

func main() {
	w := vgi.NewWorker(vgi.WithCatalogName("demo"))
	w.RegisterScalar(&UpperCaseFunction{})
	w.RunStdio()
}

use std::sync::Arc;

use arrow_array::{cast::AsArray, ArrayRef, RecordBatch, StringArray};
use arrow_schema::DataType;
use vgi::{ArgSpec, FunctionMetadata, ProcessParams, ScalarFunction, Worker};
use vgi_rpc::{Result, RpcError};

/// `upper_case(s)` — uppercase a string column.
struct UpperCase;

impl ScalarFunction for UpperCase {
    fn name(&self) -> &str {
        "upper_case"
    }

    fn metadata(&self) -> FunctionMetadata {
        FunctionMetadata {
            description: "Convert string values to uppercase".into(),
            return_type: Some(DataType::Utf8),
            ..Default::default()
        }
    }

    fn argument_specs(&self) -> Vec<ArgSpec> {
        vec![ArgSpec::column("value", 0, "varchar", "String to uppercase")]
    }

    fn process(&self, params: &ProcessParams, batch: &RecordBatch) -> Result<RecordBatch> {
        let col = batch.column(0).as_string::<i32>();
        let upper: StringArray = col.iter().map(|v| v.map(str::to_uppercase)).collect();
        let out: ArrayRef = Arc::new(upper);
        RecordBatch::try_new(params.output_schema.clone(), vec![out])
            .map_err(|e| RpcError::runtime_error(e.to_string()))
    }
}

fn main() {
    let mut worker = Worker::new();
    worker.register_scalar(UpperCase);
    worker.run(); // serves stdio (default), --unix <path>, or --http
}

import farm.query.vgi.Worker;
import farm.query.vgi.scalar.ScalarFn;
import farm.query.vgi.scalar.Vector;
import org.apache.arrow.vector.VarCharVector;

import java.nio.charset.StandardCharsets;
import java.util.Locale;

public final class DemoWorker {

    /** {@code upper_case(value: utf8) -> utf8} */
    static final class UpperCase extends ScalarFn {
        @Override public String name() { return "upper_case"; }
        @Override public String description() { return "Converts string values to uppercase"; }

        public void compute(@Vector VarCharVector value, VarCharVector result) {
            int rows = value.getValueCount();
            for (int i = 0; i < rows; i++) {
                if (value.isNull(i)) { result.setNull(i); continue; }
                byte[] up = new String(value.get(i), StandardCharsets.UTF_8)
                        .toUpperCase(Locale.ROOT)
                        .getBytes(StandardCharsets.UTF_8);
                result.setSafe(i, up, 0, up.length);
            }
        }
    }

    public static void main(String[] args) {
        Worker worker = Worker.builder()
                .catalogName("demo")
                .registerScalar(new UpperCase());
        worker.runFromArgs(args); // stdio by default; --unix / --http via flags
    }
}

In the C version, everything that isn't uppercasing is the work of knowing DuckDB: which vector holds the argument, where the nulls are tracked, and the fact that short strings sit inside the string struct while longer ones live behind a pointer. In the workers, all of that belongs to the SDK, and what's left is the part only you can write. The point isn't that C is bad — it's that this interface doesn't have to live in DuckDB's process to be the interface. All five SDKs →

Function shapes

Five shapes, matching DuckDB's own

Every function a worker exposes takes one of five shapes, and the extension registers each one in the matching DuckDB position. Those positions are DuckDB's own: scalars and aggregates appear in expressions, table functions appear in FROM, and the in-out shapes consume a relation. All that changes is where the implementation runs.

Scalar

1 row → 1 value

Transform each row independently. You receive the whole column as an Arrow array and return one — the vectorized shape, not a per-row callback.

DuckDB's scalar function (duckdb_create_scalar_function) · SELECT f(col) FROM t

Scalar worker, end to end →

Table

args → N rows

Generate rows from scalar arguments, with no input relation. Bind declares the schema; process is called repeatedly, so large results stream a bounded batch at a time.

DuckDB's table function (duckdb_create_table_function) · SELECT * FROM f(args)

Table worker, end to end →

Table-in-out

N rows → M rows

Consume a relation and stream a transformed relation back, batch by batch. Filter, enrich, reshape — nothing is held, so input size is unbounded.

DuckDB's in-out table function · SELECT * FROM f((SELECT …))

Table-in-out worker, end to end →

Aggregate

N rows → 1 value

Fold rows into per-group state, then emit a row per group. combine merges partial states, which is what lets DuckDB run your aggregate in parallel.

DuckDB's aggregate function (duckdb_create_aggregate_function) · SELECT f(col) FROM t GROUP BY k

Aggregate worker, end to end →

Buffering

stream → [state] → stream

For functions that must see every row before emitting any — a global sort, top-k, a full reduction. Sink, combine, then source the result back out.

DuckDB's in-out function with a final flush · SELECT * FROM f((SELECT …))

Buffering worker, end to end →

Each shape's hooks — what bind receives, when process is called again, how state is carried between phases — are documented per language, because the contracts are expressed in that language's type system. Python function patterns →

The transport layer

Same protocol, your choice of pipe

The SDKs own the wire, so the transport is a deployment decision rather than a code one: move a worker from a pipe to HTTP and the worker itself doesn't change — only the LOCATION you attach to does.

Benchmarks and a comparison matrix live on the dedicated RPC site. vgi-rpc.query.farm →

Distribution & hosting

Where a worker actually runs

Transports are how the bytes move. The other half of the question is how the worker gets there in the first place, and LOCATION is where you answer it: either a command DuckDB runs for you, or a URL it calls.

Once it's a URL, it's just a web service

We kept the protocol deliberately light: one request per call, Arrow IPC in the body, no HTTP/2 requirement, no trailers, and no long-lived connection to keep alive. When a call needs to be resumed, the worker hands back a continuation token rather than holding a session open, so the next call can land on a different instance and still be correct. That one property is what makes everything below possible.

The honest caveat: scale-to-zero platforms add a cold start to the first call. A catalog scan absorbs that easily; a tight scalar loop doesn't. When per-call latency matters more than idle cost, keep a warm pool behind a load balancer instead.

Catalogs & pushdown

Workers are first-class catalogs

A worker doesn't have to stop at functions — it can expose a whole database. DuckDB then plans against it with the same kind of information it would have about a local table.

Wire format

Apache Arrow IPC, all the way down

Every request and response is an Arrow IPC stream — a self-describing schema followed by record batches. That's it. There's no bespoke serialization, no JSON envelope, no language-specific wire types, and nothing to generate from a schema file. If you can read Arrow, you can implement a worker.

The full byte-level specification — opcodes, framing, error handling — is published on the RPC site.

How it compares

VGI vs. native extensions, Airport & ADBC

VGI overlaps with a few other ways of reaching data from DuckDB. Use it when you want your own code — in any language — to show up as SQL functions and tables.

Keep going

Where to next