What Haybarn gives you

Upstream issues every range read over an independent HTTP/1.1 connection: a multi-threaded scan making N range reads opens N TCP+TLS connections to the same peer, and pays a handshake for each. Haybarn negotiates h2 via ALPN and carries those reads as N concurrent streams on one connection. Measured: 4 connections down to 1 on a single-file CloudFront parquet scan, and 16 streams over 1 connection on a three-file scan. HPACK is the second win — AWS SigV4 requests carry 1–2 KB of signed headers each, and follow-up requests on the same connection compress to roughly 100 bytes, which shows up on wide scans that issue many small reads.

Negotiation alone doesn't buy multiplexing: CURLOPT_PIPEWAIT, the option that makes a transfer wait for an existing h2 connection, is a no-op inside curl_easy_perform. Real multiplexing only happens under curl_multi, which is not thread-safe — so Haybarn runs a process-global dispatcher thread that owns the multi handle, and worker threads submit to it and block on a condition variable. Caller-visible behaviour is identical to a plain synchronous perform.

When a remote file changes underneath a running read, upstream finds out afterwards: it transfers the entire requested range, then string-compares the ETag and throws. Haybarn sends the cached validator as an If-Match / If-Unmodified-Since precondition, so the server arbitrates and a mutated object comes back 412 Precondition Failed with zero body bytes. The round-trip count is identical; what you save is the whole range — 16 MiB of wasted transfer on a mutated parquet column chunk versus none.

It also removes a false positive. Upstream strips the quotes from an ETag but not the W/ weak prefix, so a server that legally alternates between W/"abc" and "abc" for the same resource — GitHub Raw does this — throws "the remote file has changed" on a file that never changed. With the comparison delegated to the server per RFC 9110 §13.1.1, strong validators are used for If-Match and weak ones fall back to If-Unmodified-Since.

Upstream disables content negotiation globally — every request goes out Accept-Encoding: identity. That is the right call for range reads, where representation flux would break byte-exact semantics, and the wrong one for everything else. Haybarn makes the policy per-method: range GET and HEAD stay identity, and every other method negotiates gzip, deflate, brotli, or zstd. Since S3 LIST responses are XML, the effect on listing-heavy workloads over a high-latency link is large.

Caller-injected Accept-Encoding headers are stripped, which closes a silent-corruption class rather than a performance one. Separately, response header names are normalized to lowercase in both HTTP clients: that is what h2 puts on the wire (RFC 9113 §8.2.1) and what AWS uses for x-amz-*, so case-sensitive consumers — the S3 region-redirect retry, and response.headers['x-amz-version-id'] in duckdb_logs_parsed('HTTP') — behave the same on either protocol version.

The retry loop already retried the transient statuses — 408, 429, 500, 503, 504 — but slept a fixed exponential backoff and ignored the server's own Retry-After header, which is the standard rate-limit signal from reverse proxies and from S3/GCS throttling. Haybarn reads it and uses it as a floor on the computed backoff, clamped to 60 seconds so a hostile or absurd value can't stall a request. Both header forms are parsed, delta-seconds and HTTP-date; with no header, behaviour is unchanged (RFC 9110 §10.2.3).

The cancellation flag also moved from the POST request info up to the base request, so an interrupted query can abort in-flight range GETs and HEADs — the requests a scan over remote parquet actually spends its time in — and not only POSTs.

Upstream uses the version in INSTALL … VERSION '…' to build the download URL and then discards it, so the pin is obeyed exactly once. Worse, the "file exists, so install is a no-op" early-out returns success while leaving the old binary in place — the pin is believed but never applied. Haybarn records the requested version as pinned_version in the install's info file and acts on it: a conflicting INSTALL errors, re-requesting the pin in effect stays idempotent, and FORCE INSTALL is the one way off. The pin is visible from SQL as a column on duckdb_extensions().

-- Pin an extension to an exact build and have the engine remember it.
INSTALL a5 FROM community VERSION '23ca175c';
LOAD a5;

SELECT extension_name, installed_version, pinned_version
FROM duckdb_extensions()
WHERE extension_name = 'a5';

-- A conflicting INSTALL now errors instead of silently no-op'ing.
INSTALL a5 FROM community VERSION 'c64c68da95';
-- Error: a5 is pinned to version '23ca175c'. Use FORCE INSTALL to change it.

-- FORCE INSTALL is the single way off a pin (or, with no VERSION, clears it).
FORCE INSTALL a5 FROM community VERSION 'c64c68da95';

pinned_version is deliberately separate from installed_version: one is what you asked for, the other is what the fetched binary reports about itself. Version strings containing path separators, whitespace, or .. are now rejected at bind time — the clause is interpolated into a download URL unescaped.

An extension installed as an npm package loads in place, with no INSTALL step and no network access at query time. Nothing is copied into the extension cache, so npm update and npm uninstall take effect immediately — and it works in read-only or containerised environments with no writable extension directory. Your lockfile becomes the record of what runs.

# Declare the extension as a normal project dependency.
npm install @haybarn/ext-avro-h1-5-5

-- No INSTALL, no network, no custom repository setting.
LOAD avro;
SELECT * FROM read_avro('events.avro');

Resolution order is local extension directories, then node_modules, then network autoinstall — so an npm-declared extension never reaches the network. The scan walks upward from two anchors: the process working directory, and the directory of the loaded Haybarn library itself, which is what makes it work for apps that embed the engine, for monorepos, and for pnpm's non-hoisted store. The binary's RSA signature is still verified at load exactly as it would be from the cache: the registry is a transport, not a new trust boundary. PyPI wheels are rolling out on the same model.

Everything Haybarn ships is built from the same engine commit, signed with the Haybarn key, and published on GitHub Releases with checksums, a detached GPG signature, and SLSA build provenance you can verify with gh attestation verify. Extensions are hosted per-asset at immutable URLs keyed by version and git sha, so a URL that worked once keeps returning the same bytes.

haybarn-unittest is worth calling out: it packages DuckDB's own SQL-logic test runner for npm and PyPI, so an extension developer can run .test files against a pinned engine build without recompiling the engine. It needed one load-bearing fix — the stock binary chdirs to a compile-time path and aborts on any other machine, even for --help.

The compatibility views stored DuckDB's internal LogicalTypeId where PostgreSQL clients expect real PG type OIDs, which broke the universal pg_attribute.atttypid = pg_type.oid join — and did it silently, because some internal IDs collide with real PG OIDs (DOUBLE is 23, and so is PG int4; DECIMAL is 21, and so is int2), so the join matched the wrong row instead of no row. Haybarn maps atttypid through the same function pg_type.oid already used, so the two agree by construction.

Two more categories came with it. NOT NULL and schema compliance on pg_type / pg_namespace — Npgsql's type loader reads every column as a non-nullable string and crashed on connect against the hardcoded NULLs. And twelve standard catalog tables that were simply absent (pg_roles, pg_extension, pg_aggregate, pg_trigger, pg_rewrite, pg_inherits, pg_language, pg_range, and others), mostly empty but present. Between them these fix connect-time failures for Npgsql (PowerBI and .NET) and introspection for pgjdbc, DBeaver, DataGrip, pgAdmin, Grafana, and QGIS.

FROM t, f(t.x, t.y, opt := 5) failed to bind. Named-parameter extraction only ran on the all-literal form, so in the column and LATERAL forms a named argument was swept into the synthesized input subquery as a phantom input column and no overload matched. Haybarn partitions the expression list first, pulling inline named parameters out before the sweep — so named arguments work in every form, not just one.

duckdb/haybarn.h defines HAYBARN as a presence flag and HAYBARN_ABI_VERSION as a monotonic counter, so extensions and language bindings can detect a Haybarn engine and gate on a divergence level in a #if. Both are wired into the C++ and C public headers. Source-level only — no struct or signature change, which is why the extension platform string is deliberately left alone.

The engine is built with native wasm exception handling and provides no legacy invoke_* JS trampolines to side modules. Upstream's Rust wasm extensions link the stock precompiled std for wasm32-unknown-emscripten, which carries legacy emscripten EH and imports invoke_*, __resumeException, __cxa_find_matching_catch and getTempRet0. The first call down an exception path resolves one of those to undefined and the query dies with an opaque TypeError: c is not a function. Install and load succeed, and simple queries succeed, because they never reach the exception path — which is what makes it so hard to diagnose.

Confirmed with wasm-objdump: the published wasm_eh binaries import 81 legacy-EH symbols for evalexpr_rhai and 15 for lindel; the threaded builds import zero. Haybarn rebuilds Rust std with native wasm EH on every variant, and the same bug class on the C++ side — vcpkg dependencies compiled without -fwasm-exceptions, so a guarded std::stoi inside PROJ throws fatally on the non-numeric OGC:CRS84 code when spatial reads Overture GeoParquet — is fixed with dedicated triplets and chainload wrapper toolchains that push the flag into every dependency.

An extension that needs to reach an authenticated API has a problem in the browser: it runs in a Worker, and an OAuth flow needs a popup on the main thread. Upstream duckdb-wasm has no way to cross that boundary. Haybarn adds six exported symbols an extension can call, plus the page-side bridges that service them.

The Worker blocks on Atomics.wait over a shared buffer while the page opens the popup and relays the authorization code back — polled on an interval so a closed popup is noticed even when no message ever arrives. Two pieces of supporting work made it possible: an _emscripten_yield override so Workers genuinely block instead of busy-spinning, and an HTTP layer rewritten to return the response body on non-2xx status, which an OAuth flow has to read to handle a 401 or a 302 at all.

import {
  composeWorkerBridges,
  installVgiOAuthBridge,
  installVgiWebWorkerBridge,
} from '@haybarn/haybarn-wasm';

// The page-side half. The extension drives OIDC discovery, PKCE, and the
// token exchange in C++; this opens the popup and relays the code back.
const onWorkerCreated = composeWorkerBridges(
  installVgiOAuthBridge,
  installVgiWebWorkerBridge({
    // Gate which worker URLs SQL may spawn. Default: same-origin only.
    resolveWorkerUrl: (loc) =>
      new URL(loc).origin === location.origin ? loc : null,
  }),
);

// Then hand it to the provider:
//   <DuckDBProvider onWorkerCreated={onWorkerCreated}>

The same bridge mechanism carries a second transport: a catalog attached with LOCATION 'worker:<url>' runs its worker client-side, exchanging Arrow batches with the extension over a duplex ring inside DuckDB's own shared linear memory. Because that means SQL can ask the page to spawn code, resolveWorkerUrl gates which URLs are allowed and defaults to same-origin only — a rejected location fails the ATTACH with a clear error rather than launching anything. Both bridges need cross-origin isolation for SharedArrayBuffer; without it they warn and no-op.

Images for linux_amd64, linux_arm64, both musl variants, and wasm are published to GHCR and tagged in lockstep with the engine release, with emscripten, vcpkg, ccache and sccache baked in at fixed absolute paths. The payoff is the compiler cache: upstream's setup-emsdk action installs to /home/runner/work/_temp/<random UUID>/emsdk-main/, so emcc's path is different on every run and nothing ever cache-hits. With emcc at /emsdk and CCACHE_BASEDIR at a stable in-container path, hashes repeat — and the roughly 300 MB emsdk download per build leg disappears with them.

The ccache version matters more than it sounds: EPEL still ships 3.7.7, from 2019, which predates the HTTP storage backend entirely, and Ubuntu 24.04's 4.9.1 has a broken bearer auth that silently dropped cache writes. 4.13.6 is a statically linked musl build, so the same binary runs on every image. Rust builds get sccache against the same shared store.

Emscripten moves from upstream's 3.1.71 to 5.0.7 — several years of wasm codegen, wasm-opt, and exception-handling work, and the reason Haybarn can build everything with native wasm EH in the first place. Alongside the base and rust images there is a third, rust-nightly, pinned to a specific nightly with the rust-src component.

That variant exists because a Rust extension for the threaded (cross-origin isolated) wasm platform needs std rebuilt with +atomics,+bulk-memory,+mutable-globals via -Z build-std, which is nightly-only. The nightly is pinned to a date rather than tracking the channel, precisely because build-std recompiles std from that toolchain's source — a floating channel would make those rebuilds irreproducible. The flags are scoped to the wasm target so host build-scripts and proc-macros are not compiled with wasm atomics.