Haybarn 2.0: DuckDB 2.0 with a much faster start in the browser

Haybarn is Query.Farm’s distribution of DuckDB: the same engine, SQL and file format, with its own signed extension builds, packages and a small set of engine changes. Haybarn 2.0 moves all of that to DuckDB 2.0.0, across the CLI, libraries, extensions and the Python, Node, Rust, Go, JDBC, ODBC and WebAssembly clients.

The current build is rc11 for the engine and native clients, and rc12 for WebAssembly, whose only change from rc11 is faster startup in the web shell. Packages keep the release candidate label for now, so the commands below use the rc channels.

To try it without installing anything, open the Haybarn shell: Haybarn-Wasm in a full-page terminal, where INSTALL and LOAD work for core and community extensions as they do on the desktop.

What Haybarn adds

Haybarn isn’t only a rebuild. It carries a set of changes on top of DuckDB, and all of them moved to 2.0.0 for this release. The main ones, each explained in more detail on How Haybarn differs:

Reading remote data. These are in Haybarn’s build of httpfs, and DuckDB’s has none of them.

• HTTP/2. A multithreaded scan carries its range reads as streams on one connection, where DuckDB opens a new HTTP/1.1 connection for each. More
• A file that changes mid-read costs no extra bytes. Range reads send the file’s ETag as an If-Match precondition, so the server answers 412 without sending the range. DuckDB downloads the range first, then compares. More
• Compression where it’s safe. Listings and API calls negotiate gzip, brotli or zstd, while range reads stay uncompressed. More
• Retry-After is honoured when a server or S3 throttles, instead of a fixed back-off. More
• Cancelling a query stops its downloads. Interrupting a query aborts the HTTP requests in flight, for every method, and any retry back-off, instead of waiting them out. More

Installs you can reproduce.

• Version pins stick. INSTALL … VERSION is recorded, and a later INSTALL or UPDATE EXTENSIONS won’t silently replace it. The pin shows as pinned_version in duckdb_extensions(). More
• Extensions from npm. An extension installed as an npm package loads straight from node_modules, with no network access at query time. More

The tools you already have.

• PostgreSQL clients connect. Fixes to pg_catalog let Npgsql, pgjdbc, DBeaver, Grafana and other PostgreSQL-protocol tools connect and introspect. More
• Named arguments work with column inputs. FROM t, f(t.x, opt := 5) failed to bind in DuckDB, because the named argument was treated as another input column. More

Bug fixes

When we started working on Haybarn 2.0, we noticed the DuckDB 2.0 release had a number of sharp edges we needed to file down, across the engine and every client. Each fix with a badge links to its upstream PR: means it’s merged, means the PR is still open, and no badge means the fix is only in Haybarn for now.

In the engine

• A clock stepping backwards could invalidate the database. Clocks can step backwards on some virtual machines, and in Safari’s pthreads. One backwards step failed the running statement with Information loss on integer cast: value -19 and invalidated the database. Elapsed intervals now clamp to zero. Upstream #28245
• The thread collecting a query’s result spun at full CPU until the workers finished, and in WebAssembly it could deadlock a LOAD. It now waits. On a query with a long tail, process CPU went from 5.99 to 4.32 seconds. With 48 threads and 48 connections, throughput rose 13 to 15%. The tradeoff: ORDER BY of 20 million rows at 48 threads got about 10% slower in median wall time. Upstream #27738
• Extensions built with MSVC could crash when loading on a Windows engine built with clang-cl, because of how the compiler pooled string literals. Haybarn’s Windows release turns that pooling off, and the build fails if the flag goes missing. The upstream issue is #27857.

Over HTTP

• Responses with a header line over 8 KiB failed. Since late October, GitHub’s release downloads send a 9.2 KB content-security-policy line, so reading a Parquet file from a GitHub release failed on the httplib backend with Failed to read connection. Haybarn raised the limit to 16 KiB, and DuckDB separately raised its own to 64 KiB. Upstream #27825

In Python

• Python 3.14 could crash or hang at exit while daemon threads were still scanning object or Arrow data. Engine threads now refuse to enter the interpreter once it’s shutting down. In our stress test, failures per 300 runs went from 13 and 39 to none on the two affected cases. DuckDB’s Python client has since fixed this separately. Upstream #719

In the browser (Wasm)

• The HTTP client’s HEAD requests logged their headers to the console, including Authorization and secrets’ extra headers. Open upstream #2257
• Header values were URL-encoded, so a quoted ETag in If-Match never matched, and a server pinning range reads answered 412. Open upstream #2297
• LOAD of a signed extension over about 4 MiB hung forever on the multithreaded bundle. Hashing the signature started one thread per MiB and ran out of pool threads.
• A worker script that failed to download hung startup. In fault injection, 6 of 6 instances hung before the fix and 0 of 20 after. Dead workers now also reject pending requests instead of leaving them waiting.
• Above a 2 GiB heap, the HTTP client read and wrote the wrong memory, because pointers were treated as signed. The upstream issue is duckdb-wasm#2258.
• TIMESTAMPTZ columns always came back in UTC, whatever SET TimeZone said. The upstream issue is duckdb-wasm#2251.

Download and use it

Pick how you’d like to use it. The commands use the release candidate channels, and we ran every example except ODBC against the published packages.

# With Node
npx haybarn@rc

# With Python
uvx haybarn-cli==2.0.0rc11

BASE=https://github.com/Query-farm-haybarn/haybarn/releases/download/haybarn-v2.0.0-rc11
curl -LO $BASE/haybarn_cli-osx-arm64.zip
curl -LO $BASE/SHA256SUMS

shasum -a 256 -c SHA256SUMS --ignore-missing
gh attestation verify haybarn_cli-osx-arm64.zip --repo Query-farm-haybarn/haybarn

unzip haybarn_cli-osx-arm64.zip
./haybarn

INSTALL spatial;
LOAD spatial;
INSTALL rapidfuzz FROM community;
LOAD rapidfuzz;

SELECT extension_name, extension_version, installed_from
FROM duckdb_extensions()
WHERE extension_name IN ('spatial', 'rapidfuzz');

FORCE INSTALL a5 FROM community VERSION '462e4d95d8';
UPDATE EXTENSIONS;
-- a5 | community | PINNED | 462e4d95d8 | 462e4d95d8

pip install --pre haybarn

import haybarn as duckdb

con = duckdb.connect()
con.install_extension("spatial")
con.load_extension("spatial")
print(con.sql("SELECT ST_AsText(ST_Point(1, 2))").fetchall())

npm install @haybarn/node-api@rc

import { DuckDBInstance } from '@haybarn/node-api';

const instance = await DuckDBInstance.create(':memory:');
const conn = await instance.connect();
const result = await conn.runAndReadAll('SELECT version() AS v, 42 AS n');
console.log(result.getRowObjects());

npm install @haybarn/haybarn-wasm@rc

import * as duckdb from '@haybarn/haybarn-wasm';

const dist = 'https://unpkg.com/@haybarn/[email protected]/dist';
const bundle = await duckdb.selectBundle({
  base: {
    mainModule: `${dist}/duckdb-base.wasm`,
    mainWorker: `${dist}/duckdb-browser-base.worker.js`,
  },
});

// A browser won't start a cross-origin worker script directly, so wrap it.
const workerUrl = URL.createObjectURL(
  new Blob([`importScripts("${bundle.mainWorker}");`], { type: 'text/javascript' }),
);
const db = new duckdb.AsyncDuckDB(new duckdb.ConsoleLogger(), new Worker(workerUrl));
await db.instantiate(bundle.mainModule);
await db.open({});

// Download, hash and compile in parallel, before the first LOAD.
await db.prefetchExtensions(['spatial']);

const conn = await db.connect();
await conn.query('LOAD spatial');
const result = await conn.query('SELECT ST_AsText(ST_Point(1, 2)) AS point');
console.log(result.toArray().map((row) => row.toJSON()));

cargo add [email protected]

use haybarn::{Connection, Result};

fn main() -> Result<()> {
    let conn = Connection::open_in_memory()?;
    let version: String = conn.query_row("SELECT version()", [], |row| row.get(0))?;
    println!("{version}");
    Ok(())
}

go get github.com/Query-farm-haybarn/haybarn-go/[email protected]

package main

import (
  "database/sql"
  "fmt"
  "log"

  _ "github.com/Query-farm-haybarn/haybarn-go/v2"
)

func main() {
  db, err := sql.Open("haybarn", "")
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  var version string
  if err := db.QueryRow("SELECT version()").Scan(&version); err != nil {
    log.Fatal(err)
  }
  fmt.Println(version)
}

<dependency>
  <groupId>farm.query.haybarn</groupId>
  <artifactId>haybarn_jdbc</artifactId>
  <version>2.0.0-rc11</version>
</dependency>

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;

public class Main {
    public static void main(String[] args) throws Exception {
        try (Connection conn = DriverManager.getConnection("jdbc:haybarn::memory:");
             Statement stmt = conn.createStatement();
             ResultSet rs = stmt.executeQuery("SELECT version()")) {
            rs.next();
            System.out.println(rs.getString(1));
        }
    }
}

Moving from DuckDB

Database files move in both directions. We wrote a file with duckdb==2.0.0, read and changed it with Haybarn, and read the changes back with DuckDB.

DuckDB-signed extensions don’t load in Haybarn by default, because Haybarn trusts its own key. Run INSTALL again to get the Haybarn builds, into their own directory at ~/.haybarn/extensions. To use another repository, add it explicitly with its own pinned key.

Coming from Haybarn 1.5, treat it as a DuckDB major version upgrade: extensions need 2.0 builds, and it’s worth testing your application against the new engine.

WebAssembly startup

We wrote about Haybarn-Wasm’s startup time on the 1.5 line. That work has moved to 2.0, and DuckDB-Wasm now has a 2.0 build, so we can compare them directly. This is the same engine version, on one machine, loading the same five extensions (icu, autocomplete, spatial, iceberg and ducklake), each from its own project’s repository. Times are from page start, and every bar uses the same scale:

That missing DuckDB-Wasm bar isn’t a slow result. DuckDB-Wasm’s multithreaded build can’t load extensions at all. Every LOAD we tried failed, including parquet, json, icu and autocomplete on their own, with could not be loaded: and no reason given. The same LOADs work on its single-threaded build and on Haybarn-Wasm’s multithreaded one. With DuckDB-Wasm, a page that needs extensions has to give up threads.

These are medians on an Apple M3, so slower machines will take longer, as the Graviton numbers in the earlier post show. Measured the same way, Haybarn-Wasm 1.5.6-rc2, from before this work began, took 2.86 seconds single-threaded and 3.46 seconds multithreaded.

Three things account for most of the difference, and the earlier post has the details:

• Lazy linking. The engine exports tens of thousands of symbols to extensions, and emscripten’s startup code walked all of them. Haybarn-Wasm resolves each one when it’s first used.
• No copies. Extensions no longer get copied onto the WebAssembly heap one byte at a time before linking.
• prefetchExtensions(). Without it, each LOAD downloads, hashes and compiles its extension before the next one starts. With it, the worker does all of that in parallel, hashes with WebCrypto, and compiles while downloading, so the LOADs themselves are short. On the single-threaded build, the five LOADs took about 170 ms after a prefetch of about 440 ms.

Prefetch is an API call you add, so upgrading alone gets you the plain Haybarn-Wasm bars. Include the extensions your extensions load: iceberg loads avro and parquet, so our prefetch list had seven names.

The Haybarn shell itself started using prefetch in rc12, and its time to the prompt went from 2.25 to 1.38 seconds on the threaded build in Chromium, and from 1.82 to 1.16 on the single-threaded build. Not every browser gains, though: Firefox stayed at 3.4 seconds. Safari doesn’t keep compiled WebAssembly between visits, so there every visit pays the full compile.

Extension availability

Haybarn builds and signs its own copy of each extension, so what matters is what you’d lose by switching. These are the differences from DuckDB’s repositories for 2.0.0, as of November 5. Anything not listed is published by both:

If you depend on something in the first two rows, stay on DuckDB for now, or open an issue so we know it matters to you.

One difference doesn’t show in that table. For DuckDB 2.0 we distribute our extensions from our own repository, so about 30 of them are stubs in DuckDB’s community repository, including its evalexpr_rhai and webmacro builds on the platforms above. In Haybarn, INSTALL rapidfuzz FROM community gets the real extension.

Some platforms differ too. DuckDB publishes extensions for macOS Intel and the WebAssembly mvp bundle, and core extensions for Windows ARM64 and Linux musl. Haybarn 2.0 doesn’t target macOS Intel, Windows ARM64 or mvp, and while its Linux musl CLIs still ship, they have only a handful of community extensions and no core ones. The Linux x86-64 CLI needs a CPU with AVX2 (x86-64-v3).

DuckDB 2.0.1 is close

DuckDB’s release calendar has the 2.0.1 merge freeze on November 11 and the release on November 18, and the v2.0-cyanoptera branch is already busy with backports. The clock, spinning and header line fixes above are on it, along with our changes to parser error offsets, SQL parameter formatting and HTTP PUT content types.

Haybarn ships on DuckDB’s cadence, so once 2.0.1 is out we’ll rebase onto it and rebuild the extensions. Patches that have landed upstream will drop out of Haybarn’s stack, which is how we want it. Three API additions for extension authors are still open upstream and stay in Haybarn until they land: runtime filter opt-out, virtual columns for DELETE RETURNING and HTTPParams::Clone().

How we measured

• Startup: medians of seven runs on an Apple M3, in headless Chromium with a fresh browser each run, with extensions from each project’s CDN. We compared @duckdb/duckdb-wasm 1.33.1-dev114.0 (DuckDB v2.0.0) with @haybarn/haybarn-wasm 2.0.0-rc12. One single-threaded prefetch run took 4.3 seconds, against 0.78 to 0.91 for the rest, and we don’t know why.
• Availability: a HEAD request for every extension on every platform, against both projects’ repositories. This shows that a file is published, not that it loads.
• Tutorials: we ran every example except ODBC against the published packages, on an Apple Silicon Mac.