Hashfuncs

Magic moment — partition a billion rows in one query

Take any column, hit it with a fast hash, modulo the partition count, and you have a stable, well-distributed bucket assignment with zero configuration:

-- Bucket every event into one of 64 shards using xxHash3
SELECT
  event_id,
  payload,
  xxh3_64(event_id) % 64 AS shard
FROM events;

xxh3_64 runs at near-RAM bandwidth on modern CPUs — typically an order of magnitude faster than a cryptographic hash. For workloads where the only “adversary” is randomness — partitioning, sharding, deduplication, Bloom filter inputs, cache keys — that’s the right trade.

Architecture

Hashfuncs is a small, focused extension: a set of scalar SQL functions, each a thin wrapper around an upstream-faithful C/C++ implementation linked into the extension binary. There is no catalog, no secrets, no network — every call is pure CPU.

Three families ship in the box:

• xxHash — Yann Collet’s family. xxh32 and xxh64 are the original 32- and 64-bit forms; xxh3_64 and xxh3_128 are the modern XXH3 variants tuned for SSE2 / AVX2 / NEON. xxh3_128_hex emits the canonical 32-character hex digest defined by the xxHash specification.
• MurmurHash3 — Austin Appleby’s classic. Three forms are exposed: murmurhash3_32, murmurhash3_128 (x86 variant), and murmurhash3_x64_128 (x64 variant). The x64 form is what Apache Cassandra’s Murmur3Partitioner and Apache Spark’s default hash partitioner use.
• RapidHash — a newer 64-bit hash optimized purely for throughput. Exposed as rapidhash; generally wins published benchmarks against XXH3_64 on x86_64.

Every function accepts any DuckDB scalar value and (optionally) a seed. Two different seeds produce two uncorrelated hash spaces from the same input — useful for double-hashing in Bloom filters or running parallel shardings without correlation.

Hashfuncs vs. crypto — picking the right extension

Both extensions output deterministic hashes, but they answer different questions.

You need…	Reach for	Why
Partitioning / sharding	xxh3_64, murmurhash3_x64_128	Speed; well-distributed for random input
Bloom / XOR / Cuckoo filter input	xxh3_64, rapidhash	UBIGINT output, near-RAM bandwidth
Cross-language fingerprint	xxh3_128_hex	Canonical hex; matches Python / Rust / C
Cassandra / Spark interop	murmurhash3_x64_128	Standard hash for those routers
Cheap deduplication	xxh3_64 or xxh3_128	64 vs 128 bits = collision-safety vs speed
Digital signatures, HMAC	crypto	Adversarial collision resistance
Tamper-evident logs / chains	crypto	SHA-2 / SHA-3 / BLAKE3
Salts, keys, nonces (CSPRNG)	crypto	crypto_random_bytes from OpenSSL
Password storage	Neither — use a real password hash (bcrypt / argon2) outside DuckDB	These are general-purpose hashes

The two extensions compose. A common pattern is a crypto digest as the durable identity hash on a row, and an xxh3_64 hash for the routing decision that picks which shard processes it.

Compared to alternatives

vs. DuckDB’s built-in hash(): hash() is non-cryptographic too and is great for join/group-by internals. But its algorithm is implementation-defined and may change between DuckDB versions — don’t persist its output, and don’t expect another tool to reproduce it. Hashfuncs gives you named, stable algorithms whose output other systems can compute identically.

vs. cryptographic hashes (SHA-256, BLAKE3): roughly an order of magnitude slower per byte, but they resist adversarial collisions. Use crypto when that property matters; don’t pay for it when it doesn’t.

vs. doing this in Python / Pandas: pulling the column out of DuckDB just to hash it usually costs more than the hash itself. xxh3_64(...) in SQL is one vectorized scan with no Python round-trip — the same algorithm xxhash would compute, but in-process.

Seeds and overloads

Every function has both an unseeded form and a seeded form. The seed type follows the hash width:

• 32-bit functions (xxh32, murmurhash3_32, murmurhash3_128) take a UINTEGER seed.
• 64-bit and 128-bit xxHash variants (xxh64, xxh3_64, xxh3_128, xxh3_128_hex) and rapidhash take a UBIGINT seed.
• murmurhash3_x64_128 takes a UINTEGER seed (matching the upstream MurmurHash3 API).

Seeded variants are the right tool whenever you want the same column to participate in two different hash spaces — for example, a Bloom filter that uses two independent hash functions, or two independent shardings with no correlation between bucket assignments.

Pairing with bitfilters

The bitfilters extension (Bloom, XOR, Quotient, and Binary Fuse filters) takes UBIGINT hashes as its input. Hashfuncs is the natural producer:

-- Build an xor8 filter from a column of strings
SELECT xor8_filter(xxh3_64(email)) AS filter
FROM allowed_users;

-- Test membership in a streaming query
SELECT *
FROM events e, filter_table f
WHERE xor8_filter_contains(f.filter, xxh3_64(e.email));

The combination gives you sub-microsecond probabilistic set-membership at SQL speed — INSTALL hashfuncs FROM community; INSTALL bitfilters FROM community; and you have everything in one DuckDB session.