Hashfuncs

Magic moment — partition a billion rows in one query

Take any column, hit it with a fast hash, modulo the partition count, and you have a stable, well-distributed bucket assignment with zero configuration:

-- Bucket every event into one of 64 shards using xxHash3
SELECT
  event_id,
  payload,
  xxh3_64(event_id) % 64 AS shard
FROM events;

xxh3_64 runs at near-RAM bandwidth on modern CPUs — typically an order of magnitude faster than a cryptographic hash. For workloads where the only “adversary” is randomness — partitioning, sharding, deduplication, Bloom filter inputs, cache keys — that’s the right trade.

Architecture

Hashfuncs is a small, focused extension: a set of scalar SQL functions, each a thin wrapper around an upstream-faithful C/C++ implementation linked into the extension binary. There is no catalog, no secrets, no network — every call is pure CPU.

Three families ship in the box:

• xxHash — Yann Collet’s family. xxh32 and xxh64 are the original 32- and 64-bit forms; xxh3_64 and xxh3_128 are the modern XXH3 variants tuned for SSE2 / AVX2 / NEON. xxh3_128_hex emits the canonical 32-character hex digest defined by the xxHash specification.
• MurmurHash3 — Austin Appleby’s classic. Three forms are exposed: murmurhash3_32, murmurhash3_128 (x86 variant), and murmurhash3_x64_128 (x64 variant). The x64 form is what Apache Cassandra’s Murmur3Partitioner and Apache Spark’s default hash partitioner use.
• RapidHash — a newer 64-bit hash optimized purely for throughput. Exposed as rapidhash; generally wins published benchmarks against XXH3_64 on x86_64.

Every function accepts any DuckDB scalar value and (optionally) a seed. Two different seeds produce two uncorrelated hash spaces from the same input — useful for double-hashing in Bloom filters or running parallel shardings without correlation.

Hashfuncs vs. crypto — picking the right extension

Both extensions output deterministic hashes, but they answer different questions.

The two extensions compose. A common pattern is a crypto digest as the durable identity hash on a row, and an xxh3_64 hash for the routing decision that picks which shard processes it.

Compared to alternatives

vs. DuckDB’s built-in hash(): hash() is non-cryptographic too and is great for join/group-by internals. But its algorithm is implementation-defined and may change between DuckDB versions — don’t persist its output, and don’t expect another tool to reproduce it. Hashfuncs gives you named, stable algorithms whose output other systems can compute identically.

vs. cryptographic hashes (SHA-256, BLAKE3): roughly an order of magnitude slower per byte, but they resist adversarial collisions. Use crypto when that property matters; don’t pay for it when it doesn’t.

vs. doing this in Python / Pandas: pulling the column out of DuckDB just to hash it usually costs more than the hash itself. xxh3_64(...) in SQL is one vectorized scan with no Python round-trip — the same algorithm xxhash would compute, but in-process.

Seeds and overloads

Every function has both an unseeded form and a seeded form. The seed type follows the hash width:

• 32-bit functions (xxh32, murmurhash3_32, murmurhash3_128) take a UINTEGER seed.
• 64-bit and 128-bit xxHash variants (xxh64, xxh3_64, xxh3_128, xxh3_128_hex) and rapidhash take a UBIGINT seed.
• murmurhash3_x64_128 takes a UINTEGER seed (matching the upstream MurmurHash3 API).

Seeded variants are the right tool whenever you want the same column to participate in two different hash spaces — for example, a Bloom filter that uses two independent hash functions, or two independent shardings with no correlation between bucket assignments.

Pairing with bitfilters

The bitfilters extension (Bloom, XOR, Quotient, and Binary Fuse filters) takes UBIGINT hashes as its input. Hashfuncs is the natural producer:

-- Build an xor8 filter from a column of strings
SELECT xor8_filter(xxh3_64(email)) AS filter
FROM allowed_users;

-- Test membership in a streaming query
SELECT *
FROM events e, filter_table f
WHERE xor8_filter_contains(f.filter, xxh3_64(e.email));

The combination gives you sub-microsecond probabilistic set-membership at SQL speed — INSTALL hashfuncs FROM community; INSTALL bitfilters FROM community; and you have everything in one DuckDB session.

Pick the right function

For digests, signatures, HMAC, secure random, or anywhere an attacker controls the input, reach for crypto. None of the functions in this extension are designed to resist adversarial collisions.

Hash partitioning

Pick a partition by taking the hash modulo the partition count:

SELECT user_id,
       xxh3_64(user_id) % 16 AS partition
FROM users;

A seed gives you a separate, uncorrelated hash space — useful when you want two different shardings of the same column without correlation between bucket assignments:

SELECT user_id,
       xxh3_64(user_id, 1) % 16 AS shard_a,
       xxh3_64(user_id, 2) % 16 AS shard_b
FROM users;

Materialize the result with CREATE TABLE AS to write a partitioned Parquet dataset.

Cross-system routing (Cassandra / Spark)

Use murmurhash3_x64_128 when DuckDB needs to compute the same routing hash that an upstream Apache Cassandra Murmur3Partitioner or Apache Spark job would assign:

-- Same hash Cassandra's Murmur3Partitioner would compute
SELECT key,
       murmurhash3_x64_128(key) AS token
FROM rows_to_route;

String cache keys

xxh3_128_hex emits the canonical 32-character hex digest used by Python xxhash and xxhash-rust — the right pick when the key needs to be a string (Redis key, filename, URL component):

-- Stable cache key for a (user, query) tuple
SELECT user_id,
       query_text,
       xxh3_128_hex(
         CAST(user_id AS VARCHAR) || '|' || query_text
       ) AS cache_key
FROM query_log;

Bloom / XOR / Cuckoo filter input

Pair with the bitfilters extension. Most filter constructors take UBIGINT hashes — exactly what xxh3_64 and rapidhash produce:

-- Build an xor8 filter from a column of strings
SELECT xor8_filter(xxh3_64(email)) AS filter
FROM allowed_users;

-- Test membership
SELECT xor8_filter_contains(f.filter, xxh3_64(:candidate_email))
       AS might_be_allowed
FROM filter_table f;

For a Bloom filter using two independent hash functions, use two seeds of the same algorithm rather than two different algorithms:

SELECT xxh3_64(key, 1) AS h1,
       xxh3_64(key, 2) AS h2
FROM data;

Cross-language fingerprints

xxh3_128_hex matches the canonical lowercase 32-character hex form (low64 || high64) defined by the xxHash specification. The same bytes hashed in any of these will produce the same digest:

SELECT xxh3_128_hex('hello');
-- c779cfaa5e523818b5e9c1ad071b3e7f

# Python — pip install xxhash (https://pypi.org/project/xxhash/)
import xxhash
xxhash.xxh3_128('hello').hexdigest()
# 'c779cfaa5e523818b5e9c1ad071b3e7f'

// Rust — xxhash-rust (https://docs.rs/xxhash-rust/latest/xxhash_rust/)
use xxhash_rust::xxh3::xxh3_128;
format!("{:032x}", xxh3_128(b"hello").swap_bytes());

Useful as a content fingerprint shared between SQL and application code.

Cheap deduplication

SELECT DISTINCT xxh3_64(...) collapses duplicate payloads at near-RAM bandwidth. Drop to 128 bits when the keyspace is large enough that 64-bit birthday collisions matter:

-- Distinct payloads in a high-volume event stream
SELECT DISTINCT xxh3_64(payload) AS payload_hash
FROM events;

-- Same idea with more headroom
SELECT DISTINCT xxh3_128(payload) AS payload_hash
FROM events;

Treat the hash as an accidental-collision-safe key, not a security claim.

Independent hashes via seeds

Two different seeds give two uncorrelated hashes from the same input. Useful for double-hashing schemes, parallel shardings, and probabilistic data structures that need multiple hash functions:

SELECT key,
       xxh3_64(key, 1) AS h1,
       xxh3_64(key, 2) AS h2,
       xxh3_64(key, 3) AS h3
FROM keys;