DuckDB Extension
Hashfuncs
Compute non-cryptographic hashes in DuckDB with xxHash, MurmurHash3, RapidHash, wyhash, komihash, SipHash, CRC-32C and FNV-1a, and reproduce BigQuery, Spark, Kafka, Iceberg and ClickHouse hashes. Use them for partitioning, row keys, cache keys, and filter inputs.
On this page
Sample data for the examples
Each browser example starts with these sample tables; sample files are shared across examples on this page. Run this setup first when trying the examples in your own DuckDB session.
-- Fixtures for the hashfuncs cookbook / technical-details examples.
--
-- The hashfuncs extension itself is pure CPU (no catalog, no network),
-- so these fixtures only exist to satisfy the *table references* the
-- documentation makes ("FROM events", "FROM users", ...). The numbers
-- chosen are tiny and deterministic so re-running the suite is stable.
--
-- Idempotent: every CREATE uses OR REPLACE.
-- The bitfilters cookbook section pairs hashfuncs with the bitfilters
-- extension. Load it best-effort so xor8_filter / xor8_filter_contains
-- resolve at parse time. If the community binary isn't available on the
-- current platform we still want the rest of the fixtures to apply.
INSTALL bitfilters FROM query_farm;
LOAD bitfilters FROM query_farm;
-- cookbook.mdx โ "Hash partitioning" and "Independent hashes via seeds"
CREATE OR REPLACE TABLE users AS
SELECT i AS user_id
FROM range(0, 8) t(i);
-- cookbook.mdx โ "Cross-system routing (Cassandra / Spark)"
CREATE OR REPLACE TABLE rows_to_route AS
SELECT 'k' || i::VARCHAR AS key
FROM range(0, 4) t(i);
-- cookbook.mdx โ "String cache keys"
CREATE OR REPLACE TABLE query_log AS
SELECT i AS user_id,
'select ' || i::VARCHAR AS query_text
FROM range(0, 4) t(i);
-- cookbook.mdx โ "Bloom / XOR / Cuckoo filter input"
CREATE OR REPLACE TABLE allowed_users AS
SELECT 'user' || i::VARCHAR || '@example.com' AS email
FROM range(0, 4) t(i);
-- cookbook.mdx โ second bitfilters block reads from `filter_table f`
-- (a one-row table holding a pre-built xor8 filter).
CREATE OR REPLACE TABLE filter_table AS
SELECT xor8_filter(xxh3_64(email)) AS filter
FROM allowed_users;
-- cookbook.mdx โ "Independent hashes via seeds" uses `FROM keys`
CREATE OR REPLACE TABLE keys AS
SELECT 'k' || i::VARCHAR AS key
FROM range(0, 4) t(i);
-- cookbook.mdx + technical-details.mdx โ "events" with event_id, payload
-- Used by sharding / dedup / "magic moment" examples.
CREATE OR REPLACE TABLE events AS
SELECT i AS event_id,
'evt-' || i::VARCHAR AS payload,
'user' || i::VARCHAR || '@example.com' AS email
FROM range(0, 8) t(i);
-- technical-details.mdx โ "double-hash with two seeds" reads `FROM data`
CREATE OR REPLACE TABLE data AS
SELECT 'k' || i::VARCHAR AS key
FROM range(0, 4) t(i);
Create repeatable hash values for data processing
Hashfuncs exposes named non-cryptographic algorithms such as xxHash, MurmurHash, and RapidHash. They turn a value into a fixed-size number that can be used to assign partitions, construct lookup keys, or compare fingerprints across systems.
Choose a specific algorithm, seed, and input representation when another program must reproduce the result. Hash collisions are possible, so an equal hash alone does not establish that two records are equal. For cryptographic digests or HMACs, use Crypto.
Example
Assign keys to hash partitions
Hash each sample key with the same seed and map it to one of four partition numbers.
Tested with DuckDB 1.5.6. Browser examples use Haybarn-Wasm 2.0.0-rc12.
SET allow_extension_repositories = 'allowed';
CREATE EXTENSION REPOSITORY IF NOT EXISTS query_farm
WITH PREFIX 'https://duckdb-extensions.query.farm';
INSTALL hashfuncs FROM query_farm;
LOAD hashfuncs FROM query_farm;
WITH users(user_id) AS (VALUES ('alice'), ('bob'), ('carol'))
SELECT user_id, xxh3_64(user_id, 42) % 4 AS partition_id
FROM users
ORDER BY user_id;
SET allow_extension_repositories = 'allowed';
CREATE EXTENSION REPOSITORY IF NOT EXISTS query_farm
WITH PREFIX 'https://haybarn-extensions.query.farm';
INSTALL hashfuncs FROM query_farm;
LOAD hashfuncs FROM query_farm;
WITH users(user_id) AS (VALUES ('alice'), ('bob'), ('carol'))
SELECT user_id, xxh3_64(user_id, 42) % 4 AS partition_id
FROM users
ORDER BY user_id;
INSTALL hashfuncs FROM community;
LOAD hashfuncs;
WITH users(user_id) AS (VALUES ('alice'), ('bob'), ('carol'))
SELECT user_id, xxh3_64(user_id, 42) % 4 AS partition_id
FROM users
ORDER BY user_id;
Expected result
| user_id | partition_id |
|---|---|
| alice | 2 |
| bob | 2 |
| carol | 3 |
Keep the algorithm, input type, and seed consistent wherever you compute the partition. These hashes are not suitable for passwords or message authentication.
Common tasks
- Choose a hash function Compare the available algorithms and output widths.
- Partition a table Assign rows to a fixed number of buckets.
- Match hashes across languages Check byte order and build compatibility.
- Hash several columns Build one unambiguous key from a row.
- Match another system Reproduce BigQuery, Spark, Kafka and Iceberg hashes.
Deep Dive
Technical Details
These examples use DuckDB 2.0 or later. Complete the repository setup and installation first.
Magic moment โ partition a billion rows in one query
Take any column, hit it with a fast hash, modulo the partition count, and you have a stable, well-distributed bucket assignment with zero configuration:
-- Bucket every event into one of 64 shards using xxHash3SELECT event_id, payload, xxh3_64(event_id) % 64 AS shardFROM events;xxh3_64 runs at near-RAM bandwidth on modern CPUs โ typically an order of magnitude faster than a cryptographic hash. For workloads where the only โadversaryโ is randomness โ partitioning, sharding, deduplication, Bloom filter inputs, cache keys โ thatโs the right trade.
Every function in this extension is a non-cryptographic hash. They are fast and well-distributed against random inputs, but they do not resist a motivated adversary who controls the input. Treating xxh3_128 like a SHA-256 digest is a security bug, not a performance optimization.
For digital signatures, HMAC, content-addressed storage with adversarial input, password salts, tamper-evident audit logs, or secure random bytes, use the crypto extension instead โ it provides BLAKE3, SHA-2, SHA-3, HMAC, and a CSPRNG.
Architecture
Hashfuncs is a small, focused extension: a set of scalar SQL functions, each a thin wrapper around an upstream-faithful C/C++ implementation linked into the extension binary. There is no catalog, no secrets, no network โ every call is pure CPU.
What ships in the box:
- xxHash โ Yann Colletโs family.
xxh32andxxh64are the original 32- and 64-bit forms;xxh3_64andxxh3_128are the modern XXH3 variants tuned for SSE2 / AVX2 / NEON.xxh3_128_hexemits the canonical 32-character hex digest defined by the xxHash specification. - MurmurHash3 โ Austin Applebyโs classic. Three standard forms are exposed:
murmurhash3_32,murmurhash3_128(x86 variant), andmurmurhash3_x64_128(x64 variant). - RapidHash โ a newer 64-bit hash optimized purely for throughput. Exposed as
rapidhash; generally wins published benchmarks against XXH3_64 on x86_64.rapidhash_microandrapidhash_nanoare its compact upstream variants. - wyhash and komihash โ compact, very fast 64-bit hashes common in hash tables and language runtimes:
wyhashandkomihash. - SipHash โ keyed hashing that resists hash-flooding when the key is secret:
siphash24andsiphash13. - Checksums and classic hashes โ
crc32c(CRC-32C, extendable across pieces),fnv1a_32andfnv1a_64. - Exact matches for other systems โ
farm_fingerprint(BigQuery),spark_hashandspark_xxhash64(Spark),kafka_murmur2andkafka_partition(Kafka),iceberg_bucketandiceberg_hash(Iceberg), andclickhouse_cityhash64(ClickHouse). See Matching other systems. - Consistent hashing โ
jump_hashassigns keys to shards so that adding a shard moves as few keys as possible. See Consistent hashing for resharding.
Every hash function accepts any supported value โ including whole rows, structs and lists โ and (optionally) a seed. Two different seeds produce two uncorrelated hash spaces from the same input โ useful for double-hashing in Bloom filters or running parallel shardings without correlation.
Hashfuncs vs. crypto โ picking the right extension
Both extensions output deterministic hashes, but they answer different questions.
| You needโฆ | Reach for | Why |
|---|---|---|
| Partitioning / sharding | xxh3_64, murmurhash3_x64_128 |
Speed; well-distributed for random input |
| Bloom / XOR / Cuckoo filter input | xxh3_64, rapidhash |
UBIGINT output, near-RAM bandwidth |
| Cross-language fingerprint | xxh3_128_hex |
Canonical hex; matches Python / Rust / C |
| Multi-column row keys | xxh3_64 / xxh3_128 of a tuple |
Unambiguous row encoding |
| Same hash as BigQuery / Spark / Kafka / Iceberg / ClickHouse | farm_fingerprint, spark_hash, kafka_partition, iceberg_bucket, clickhouse_cityhash64 |
Tested against each system |
| Resharding with minimal data movement | jump_hash |
Adding a shard moves ~1/(n + 1) of keys |
| Hash-flooding resistance (secret key) | siphash24, siphash13 |
Keyed; still not a MAC |
| Cheap deduplication | xxh3_64 or xxh3_128 |
64 vs 128 bits = collision-safety vs speed |
| Digital signatures, HMAC | crypto |
Adversarial collision resistance |
| Tamper-evident logs / chains | crypto |
SHA-2 / SHA-3 / BLAKE3 |
| Salts, keys, nonces (CSPRNG) | crypto |
crypto_random_bytes from OpenSSL |
| Password storage | Neither โ use a real password hash (bcrypt / argon2) outside DuckDB | These are general-purpose hashes |
The two extensions compose. A common pattern is a crypto digest as the durable identity hash on a row, and an xxh3_64 hash for the routing decision that picks which shard processes it.
Compared to alternatives
vs. DuckDBโs built-in hash(): hash() is non-cryptographic too and is great for join/group-by internals. But its algorithm is implementation-defined and may change between DuckDB versions โ donโt persist its output, and donโt expect another tool to reproduce it. Hashfuncs gives you named, stable algorithms whose output other systems can compute identically.
vs. cryptographic hashes (SHA-256, BLAKE3): roughly an order of magnitude slower per byte, but they resist adversarial collisions. Use crypto when that property matters; donโt pay for it when it doesnโt.
vs. doing this in Python / Pandas: pulling the column out of DuckDB just to hash it usually costs more than the hash itself. xxh3_64(...) in SQL is one vectorized scan with no Python round-trip โ the same algorithm xxhash would compute, but in-process.
Seeds and overloads
Every hash function has an unseeded form and a seeded form, except FNV-1a, which has no seed. The seed type follows the upstream API:
xxh32,murmurhash3_32,murmurhash3_128andmurmurhash3_x64_128take aUINTEGERseed (matching the upstream MurmurHash3 API).xxh64,xxh3_64,xxh3_128,xxh3_128_hex, the RapidHash variants,wyhashandkomihashtake aUBIGINTseed.siphash24andsiphash13take a 128-bitkeyasUHUGEINT; its 16 little-endian bytes are the SipHash key. Without a key, all key bytes are zero.crc32cโs second argument is the CRC of the data before this value, socrc32c(b, crc32c(a))is the checksum ofafollowed byb.
The unseeded form always equals the seeded form with seed 0.
Seeded variants are the right tool whenever you want the same column to participate in two different hash spaces โ for example, a Bloom filter that uses two independent hash functions, or two independent shardings with no correlation between bucket assignments.
Input types โ what gets hashed
The hash functions hash a valueโs bytes, so it matters which bytes those are:
| Input | Bytes hashed |
|---|---|
VARCHAR, BLOB |
The stringโs UTF-8 bytes or the blobโs bytes |
Integers, FLOAT, DOUBLE, BOOLEAN |
The valueโs little-endian in-memory bytes โ so the width matters: xxh64(42::INTEGER) and xxh64(42::BIGINT) differ |
DECIMAL |
The stored integer (unscaled value), whose width follows the precision |
DATE, TIME, TIMESTAMP (all precisions and WITH TIME ZONE) |
The stored count (days, microseconds, โฆ) |
INTERVAL |
Months, days and microseconds as stored โ INTERVAL '1 day' and INTERVAL '24 hours' hash differently |
UUID |
Its 16 bytes in standard (RFC 4122) order, matching a hash of the UUIDโs binary form elsewhere |
ENUM |
Its label, so it hashes like the same VARCHAR |
STRUCT, LIST, MAP, ARRAY, tuples |
The row encoding below |
NULL |
Returns NULL |
0.0 and -0.0 have different bytes and so hash differently, as do NaNs with different bit patterns. The functions deliberately donโt normalize them, so hashes already stored stay valid. When values that compare equal in SQL must hash equal, normalize first:
SELECT xxh64(CASE WHEN isnan(x) THEN 'NaN'::DOUBLE WHEN x = 0 THEN 0.0::DOUBLE ELSE x END) AS hFROM (VALUES (-0.0::DOUBLE), (0.0::DOUBLE)) t(x);For the same reason, cast integer inputs to one width (for example BIGINT) wherever the hash must agree across tables or systems.
Types without a stable byte representation (BIT, UNION, VARIANT, โฆ) are rejected when the query is planned, with an error naming the function and type โ never halfway through a query.
Hashing several columns at once
Pass a tuple โ xxh3_64((customer_id, order_date, sku)) โ or any STRUCT, LIST, MAP or ARRAY, and the whole value is hashed through one encoding. This is the right way to build multi-column keys: concatenating strings is slower and collides ('a|' || 'b' equals 'a' || '|b'), while the encoding below never does for values of the same type.
-- One 64-bit key per (event_id, payload) pairSELECT event_id, payload, xxh3_64((event_id, payload)) AS row_keyFROM events;The encoding is stable โ stored row hashes stay valid across releases โ and simple enough to reproduce in other languages:
| Part | Encoding |
|---|---|
| Field | 0x00 if NULL; otherwise 0x01 followed by the value |
VARCHAR, BLOB |
Length as a little-endian 64-bit integer, then the bytes |
LIST, ARRAY |
Element count as a little-endian 64-bit integer, then each element as a field |
MAP |
A list of (key, value) structs |
STRUCT, tuple |
Each child as a field, in order; field names are not encoded |
UUID |
Its 16 RFC 4122 bytes |
| Anything else | The same bytes it hashes as on its own |
So xxh64((1, 'ab')) hashes the bytes 01 01000000 01 0200000000000000 6162. An ARRAY hashes the same as a LIST with the same elements, and {'x': 1, 'y': 2} hashes the same as (1, 2). A NULL row returns NULL; a row whose fields are all NULL does not.
Matching other systems
These functions reproduce another systemโs hash exactly โ the same algorithm, byte encoding, seed and signed or unsigned result. Their tests compare against values produced by the real systems: BigQueryโs documented examples, Spark 4.2.1, Kafkaโs own partitioner code (kafka-clients 4.4.0), the Iceberg specโs test values and Icebergโs own Java library (1.12.0), and ClickHouse.
| System | Their function | Hashfuncs |
|---|---|---|
| BigQuery | FARM_FINGERPRINT(x) |
farm_fingerprint(x) |
| Spark | hash(a, b, ...) |
spark_hash(a, b, ...) |
| Spark | xxhash64(a, b, ...) |
spark_xxhash64(a, b, ...) |
| Spark bucketing | pmod(hash(cols), n) |
((spark_hash(cols) % n) + n) % n |
| Kafka default partitioner | partition of a keyed record | kafka_partition(key, n) |
| Iceberg | bucket[N] partition transform |
iceberg_bucket(value, n) |
| ClickHouse | cityHash64(s) |
clickhouse_cityhash64(s) |
| ClickHouse | sipHash64(s) |
siphash24(s) |
-- BigQuery's documented FARM_FINGERPRINT example, in DuckDBSELECT farm_fingerprint('1footrue') AS row_fingerprint;-- -1541654101129638711
-- Spark's documented hash() exampleSELECT spark_hash('Spark', [123], 2);-- -1321691492A few rules to know:
- Spark hashes values by its own type rules, not by DuckDBโs bytes:
BOOLEAN,TINYINT,SMALLINT,INTEGERandDATEas a 32-bit int,BIGINTand timestamps as a 64-bit long (microseconds โ DuckDBTIMESTAMPcorresponds to SparkโsTIMESTAMP_NTZandTIMESTAMPTZto SparkโsTIMESTAMP),-0.0as0.0,DECIMALby its unscaled value, and lists, arrays and structs element by element. Each argumentโs hash seeds the next, starting from 42, andNULLs are skipped. Sparkโs string hashing mixes trailing bytes non-standardly, somurmurhash3_32does not match Spark โ usespark_hash. Types Spark doesnโt have (unsigned integers,UUID,MAP, โฆ) are rejected; cast them to a Spark type first. - Kafka hashes the record keyโs serialized bytes. A
VARCHARis hashed as UTF-8, which matches KafkaโsStringSerializer; pass aBLOBfor keys serialized another way. Records without a key arenโt hashed by Kafka, so aNULLkey returnsNULL. - Iceberg hashes each type by its spec encoding, not DuckDBโs bytes: every integer width as an 8-byte long (so
34and34::BIGINTshare a bucket),DECIMALby its unscaled value regardless of scale, dates as days, and times and timestamps as microseconds โ nanosecond timestamps are floored to microseconds first. Iceberg canโt bucketBOOLEAN,FLOATorDOUBLE, so those are rejected. - ClickHouse hashes numbers and multiple arguments differently from strings, so the ClickHouse functions take a single
VARCHARorBLOB. - Cassandraโs
Murmur3Partitioneris derived from MurmurHash3 x64-128 but its token is not the rawmurmurhash3_x64_128value; verify against Cassandra before relying on it for routing.
Consistent hashing for resharding
hash % n reassigns almost every key when n changes, so growing a cluster from 10 to 11 shards moves about 90% of the data. jump_hash implements Jump consistent hash: going from n to n + 1 buckets moves only about 1/(n + 1) of the keys, and every key that moves goes to the new bucket. It returns the same buckets as Guavaโs Hashing.consistentHash.
-- How many of 100,000 keys move when an 11th shard is added?SELECT count(*) FILTER (WHERE xxh3_64(i) % 11 <> xxh3_64(i) % 10) AS moved_with_modulo, count(*) FILTER (WHERE jump_hash(xxh3_64(i), 11) <> jump_hash(xxh3_64(i), 10)) AS moved_with_jump_hashFROM range(100000) t(i);Hash the value first โ jump_hash takes a 64-bit key, and buckets are numbered 0 to n - 1. Jump hash suits shards that are numbered and only added or removed at the end; for removing arbitrary nodes from the middle, use a ring or rendezvous scheme instead.
Pairing with bitfilters
The bitfilters extension (Bloom, XOR, Quotient, and Binary Fuse filters) takes UBIGINT hashes as its input. Hashfuncs is the natural producer:
-- Build an xor8 filter from a column of stringsSELECT xor8_filter(xxh3_64(email)) AS filterFROM allowed_users;-- Test membership in a streaming querySELECT *FROM events e, filter_table fWHERE xor8_filter_contains(f.filter, xxh3_64(e.email));The combination gives you sub-microsecond probabilistic set-membership at SQL speed. After installing and loading Hashfuncs, add Bitfilters with INSTALL bitfilters FROM query_farm; LOAD bitfilters FROM query_farm; to use both in one DuckDB session.
XXH3-128 compatibility across builds
The October 7, 2026 v1.5 source fix corrects xxh3_128_hex to the canonical high64 then low64 order for both strings and blobs, with or without a seed. The DuckDB 2.0 builds in the Query.Farm repository include it. Community packages for DuckDB 1.5 may lag this source change. Check the installed behavior before exchanging digests:
SELECT xxh3_128_hex('hello') AS digest;-- Corrected: b5e9c1ad071b3e7fc779cfaa5e523818-- Older builds (including 0dec806): c779cfaa5e523818b5e9c1ad071b3e7fPreviously stored hex digests from affected builds need their two 16-character halves exchanged once to match the corrected output. Do not keep applying that workaround to corrected digests.
The numeric xxh3_128 functions retain their historical UHUGEINT layout so existing numeric keys remain stable. Their printed hex is not canonical: convert it explicitly when interchanging with another implementation.
WITH h AS ( SELECT lower(printf('%032x', xxh3_128('hello'))) AS legacy_hex)SELECT substr(legacy_hex, 17, 16) || substr(legacy_hex, 1, 16) AS canonical_hexFROM h;-- b5e9c1ad071b3e7fc779cfaa5e523818Reference
Extension Contents
Quick reference to all available functions and settings organized by category.
| Name | Type | Description |
|---|---|---|
|
CRC-32C & FNV-1a
Classic checksum and hash formats that other systems embed: CRC-32C (Castagnoli), used by iSCSI, ext4, Google Cloud Storage object checksums and many storage formats, and FNV-1a, a tiny byte-at-a-time hash common in older code. |
||
| crc32c() | Object type: Scalar function |
CRC-32C (Castagnoli) checksum as UINTEGER โ the checksum used by iSCSI, ext4, Google Cloud Storage objects and many storage and wire formats.
|
| fnv1a_32() | Object type: Scalar function | 32-bit FNV-1a. |
| fnv1a_64() | Object type: Scalar function | 64-bit FNV-1a. |
|
Consistent hashing
Assign keys to a number of buckets so that changing the bucket count moves as few keys as possible. |
||
| jump_hash() | Object type: Scalar function |
Jump consistent hash (Lamping and Veach): assigns a 64-bit key to one of buckets buckets so that growing from n to n + 1 buckets moves only about 1/(n + 1) of the keys โ all of them into the new bucket โ where hash % n would move almost every key.
|
|
Matching other systems
Functions that reproduce another system's hash exactly โ same algorithm, same byte encoding, same seed and same signed or unsigned result โ so DuckDB can compute keys, buckets and partitions that line up with BigQuery, Apache Spark, Apache Kafka, Apache Iceberg and ClickHouse. Each is tested against values produced by the real system. |
||
| clickhouse_cityhash64() | Object type: Scalar function |
ClickHouse's cityHash64 of a String: CityHash64 from CityHash v1.0.2, the version ClickHouse ships, as UBIGINT.
|
| farm_fingerprint() | Object type: Scalar function |
FarmHash Fingerprint64 as a signed BIGINT โ the same value as BigQuery's FARM_FINGERPRINT for the same STRING or BYTES.
|
| iceberg_bucket() | Object type: Scalar function |
The partition Apache Iceberg's bucket[N] transform assigns a value: (murmur3_x86_32(value) & Integer.MAX_VALUE) % N, using the Iceberg spec's encoding of each type.
|
| iceberg_hash() | Object type: Scalar function |
Iceberg's 32-bit bucket hash (spec Appendix B) as a signed INTEGER: MurmurHash3 x86_32 with seed 0 over Iceberg's encoding of the value โ integers as 8-byte little-endian longs, DECIMAL as the minimal big-endian bytes of its unscaled value, dates as days, times and timestamps as microseconds (nanosecond timestamps floored), strings as UTF-8 and UUIDs in big-endian order.
|
| kafka_murmur2() | Object type: Scalar function |
Kafka's murmur2 hash of a serialized record key (VARCHAR as UTF-8 bytes, or BLOB), as a signed INTEGER โ the hash Kafka's default partitioner applies to keyed records.
|
| kafka_partition() | Object type: Scalar function |
The partition Kafka's default partitioner assigns a keyed record: toPositive(murmur2(key)) % partitions.
|
| spark_hash() | Object type: Scalar function |
Spark's hash(): Murmur3 x86_32 with seed 42, returning a signed INTEGER.
|
| spark_xxhash64() | Object type: Scalar function |
Spark's xxhash64(): XXH64 with seed 42, returning a signed BIGINT, with the same multi-value seeding, NULL skipping and type rules as spark_hash.
|
|
MurmurHash3
Austin Appleby's MurmurHash3 โ a battle-tested non-cryptographic hash used across many distributed systems. Choose |
||
| murmurhash3_128() | Object type: Scalar function | 128-bit MurmurHash3 โ the x86 variant. |
| murmurhash3_32() | Object type: Scalar function | 32-bit MurmurHash3. |
| murmurhash3_x64_128() | Object type: Scalar function | 128-bit MurmurHash3 โ the x64 variant. |
|
RapidHash
RapidHash is purpose-built for raw 64-bit throughput while keeping competitive distribution quality. Frequently the fastest published 64-bit non-cryptographic hash on x86_64 โ reach for |
||
| rapidhash() | Object type: Scalar function | RapidHash โ a 64-bit non-cryptographic hash designed for exceptional speed while keeping competitive distribution quality. |
| rapidhash_micro() | Object type: Scalar function | RapidHash Micro โ the upstream compact RapidHash variant with a smaller code footprint. |
| rapidhash_nano() | Object type: Scalar function | RapidHash Nano โ the smallest upstream RapidHash variant. |
|
SipHash
SipHash is a keyed hash: with a secret 128-bit key, an attacker who doesn't know the key can't craft colliding inputs, which protects hash tables and partitioners from hash-flooding. |
||
| siphash13() | Object type: Scalar function | SipHash-1-3 โ the faster, reduced-round SipHash used by Rust's standard library and CPython 3.11+ for string hashing. |
| siphash24() | Object type: Scalar function |
SipHash-2-4, the standard SipHash, returning the 64-bit result as UBIGINT.
|
|
wyhash & komihash
Two compact, very fast 64-bit hashes popular in hash-table and language-runtime implementations: wyhash ( |
||
| komihash() | Object type: Scalar function |
64-bit komihash, version 5 โ a fast, high-quality hash whose output has been stable since 5.0 and matches hash4j's komihash5_0.
|
| wyhash() | Object type: Scalar function |
64-bit wyhash, final version 4 with its original default secret โ the version implemented by Java's hash4j (wyhashFinal4).
|
|
xxHash
Yann Collet's xxHash family โ among the fastest non-cryptographic hashes available, with excellent distribution. |
||
| xxh3_128() | Object type: Scalar function |
128-bit xxHash3 (XXH3_128) returned as a UHUGEINT.
|
| xxh3_128_hex() | Object type: Scalar function |
128-bit xxHash3 returned as a 32-character lowercase hex string in canonical xxHash byte order (high64 || low64), as defined by the xxHash specification.
|
| xxh3_64() | Object type: Scalar function | 64-bit xxHash3 (XXH3_64) โ the modern xxHash variant tuned for vectorized CPUs (SSE2/AVX2). |
| xxh32() | Object type: Scalar function | 32-bit xxHash (XXH32) โ the classic 32-bit member of the xxHash family. |
| xxh64() | Object type: Scalar function | 64-bit xxHash (XXH64). |
No extension contents match that search.
API Reference
Function Reference
Practical Examples
Cookbook
Real-world recipes and patterns for common use cases.
Pick the right function
| You wantโฆ | Pick |
|---|---|
| A modern fast 64-bit default | xxh3_64 |
| Maximum throughput | rapidhash |
| One key for several columns | xxh3_64 of a tuple: xxh3_64((a, b, c)) |
| The same hash BigQuery, Spark, Kafka, Iceberg or ClickHouse computes | farm_fingerprint, spark_hash, kafka_partition, iceberg_bucket, clickhouse_cityhash64 |
| Shards you can add to without moving most keys | jump_hash |
| A checksum another system verifies | crc32c |
| Stable hex digest matching Python / Rust xxhash | xxh3_128_hex |
| 128-bit space for billions of distinct keys | xxh3_128 or murmurhash3_x64_128 |
| Cryptographic guarantees | not this extension โ use crypto |
For digests, signatures, HMAC, secure random, or anywhere an attacker controls the input, reach for crypto. None of the functions in this extension are designed to resist adversarial collisions.
Hash partitioning
Pick a partition by taking the hash modulo the partition count:
SELECT user_id, xxh3_64(user_id) % 16 AS partitionFROM users;A seed gives you a separate, uncorrelated hash space โ useful when you want two different shardings of the same column without correlation between bucket assignments:
SELECT user_id, xxh3_64(user_id, 1) % 16 AS shard_a, xxh3_64(user_id, 2) % 16 AS shard_bFROM users;Materialize the result with CREATE TABLE AS to write a partitioned Parquet dataset.
Multi-column keys
Hash a tuple to get one key for several columns. Unlike concatenating strings, the row encoding canโt confuse ('a|', 'b') with ('a', '|b'), and it keeps NULL apart from an empty string:
-- One 64-bit key per (user, query) pair, e.g. for deduplication or joinsSELECT user_id, query_text, xxh3_64((user_id, query_text)) AS row_keyFROM query_log;Any STRUCT, LIST, MAP or ARRAY column works the same way, and every algorithm accepts a tuple โ rapidhash((a, b), 7), xxh3_128((a, b, c)).
Routing like Kafka, Spark, Iceberg and BigQuery
Predict which partition a keyed Kafka record lands on โ the same answer as Kafkaโs default partitioner โ for example to co-partition DuckDB output with a topic:
-- Partition for each key on a 12-partition topicSELECT key, kafka_partition(key, 12) AS partitionFROM rows_to_route;Reproduce a Spark bucket id. Spark buckets rows by pmod(hash(cols), numBuckets); spark_hash applies Sparkโs type rules, so pass the same column types Spark sees:
-- Spark bucket id for 16 bucketsSELECT key, ((spark_hash(key) % 16) + 16) % 16 AS spark_bucketFROM rows_to_route;Compute the Iceberg bucket[N] partition of each row โ for example to find which bucket files hold a key, or to write data partitioned the way an Iceberg table expects:
-- Iceberg bucket[16] partition of each keySELECT key, iceberg_bucket(key, 16) AS bucketFROM rows_to_route;Reproduce a BigQuery sampling split built on FARM_FINGERPRINT:
-- Same rows BigQuery's MOD(ABS(FARM_FINGERPRINT(key)), 10) = 0 would keepSELECT keyFROM rows_to_routeWHERE abs(farm_fingerprint(key)) % 10 = 0;For Apache Cassandra, note that Murmur3Partitioner tokens are derived from MurmurHash3 x64-128 but are not the raw murmurhash3_x64_128 value โ verify against Cassandra before routing with it.
Adding shards without moving every key
With hash % n, adding a shard reassigns almost every key. jump_hash moves only the keys the new shard takes over:
-- Which keys move when growing from 4 to 5 shards?SELECT key, jump_hash(xxh3_64(key), 4) AS shard_before, jump_hash(xxh3_64(key), 5) AS shard_afterFROM rows_to_route;Every key either keeps its shard or moves to the new shard 4; about a fifth of them move.
String cache keys
xxh3_128_hex emits the canonical 32-character hex digest used by Python xxhash and xxhash-rust โ the right pick when the key needs to be a string (Redis key, filename, URL component):
-- Stable cache key for a (user, query) tupleSELECT user_id, query_text, xxh3_128_hex( CAST(user_id AS VARCHAR) || '|' || query_text ) AS cache_keyFROM query_log;Concatenation is only unambiguous when the separator canโt appear in the values before it โ true here, since user_id is a number. When the key doesnโt need to be a string, hash the tuple instead (see Multi-column keys).
Bloom / XOR / Cuckoo filter input
Pair with the bitfilters extension. Most filter constructors take UBIGINT hashes โ exactly what xxh3_64 and rapidhash produce:
-- Build an xor8 filter from a column of stringsSELECT xor8_filter(xxh3_64(email)) AS filterFROM allowed_users;-- Test membership AS might_be_allowedFROM filter_table f;For a Bloom filter using two independent hash functions, use two seeds of the same algorithm rather than two different algorithms:
SELECT xxh3_64(key, 1) AS h1, xxh3_64(key, 2) AS h2FROM data;Cross-language fingerprints
These examples require the corrected v1.5 source from October 7, 2026. Older community builds may still emit the reversed halves; see XXH3-128 compatibility across builds before migrating stored digests.
xxh3_128_hex matches the canonical lowercase 32-character hex form (high64 || low64) defined by the xxHash specification. The same bytes hashed in any of these will produce the same digest:
SELECT xxh3_128_hex('hello');-- b5e9c1ad071b3e7fc779cfaa5e523818# Python โ pip install xxhash (https://pypi.org/project/xxhash/)import xxhashxxhash.xxh3_128(b'hello').hexdigest()# 'b5e9c1ad071b3e7fc779cfaa5e523818'// Rust โ xxhash-rust (https://docs.rs/xxhash-rust/latest/xxhash_rust/)use xxhash_rust::xxh3::xxh3_128;format!("{:032x}", xxh3_128(b"hello"));Useful as a content fingerprint shared between SQL and application code.
Cheap deduplication
SELECT DISTINCT xxh3_64(...) collapses duplicate payloads at near-RAM bandwidth. Drop to 128 bits when the keyspace is large enough that 64-bit birthday collisions matter:
-- Distinct payloads in a high-volume event streamSELECT DISTINCT xxh3_64(payload) AS payload_hashFROM events;-- Same idea with more headroomSELECT DISTINCT xxh3_128(payload) AS payload_hashFROM events;Treat the hash as an accidental-collision-safe key, not a security claim.
Independent hashes via seeds
Two different seeds give two uncorrelated hashes from the same input. Useful for double-hashing schemes, parallel shardings, and probabilistic data structures that need multiple hash functions:
SELECT key, xxh3_64(key, 1) AS h1, xxh3_64(key, 2) AS h2, xxh3_64(key, 3) AS h3FROM keys;Platform Support
Compatibility
Extension availability may vary by platform and DuckDB version. Check below to ensure this extension supports your environment before installation.
Quick Facts
Platforms
DuckDB 2.0.0
- Linux x86_64aarch64
- Linux (musl) Not available
- macOS Apple Silicon
- Windows x86_64x86_64 (MinGW)
- WASM Not available
Compiled binary sizes
| Platform | Architecture | Size |
|---|---|---|
| Linux | x86_64 | 4.00 MB |
| Linux | aarch64 | 3.36 MB |
| macOS | Apple Silicon | 2.83 MB |
| Windows | x86_64 | 10.23 MB |
| Windows | x86_64 (MinGW) | 17.85 MB |
Compressed download sizes for the release shown above.
DuckDB & Haybarn
Release calendar- DuckDB v2.0.0 Query.Farm repository Supported
- Haybarn v1.5.6 Built-in community repository Supported
Browser examples use Haybarn-Wasm 2.0.0-rc12.
Checked .
Learn and explore
Tutorials and articles
Writing about Hashfuncs: how it works, what changed, and why.
-
Tutorial ยท 7 min ยท
Assign Users to Repeatable Experiment Groups with DuckDB
Use a named hash algorithm and fixed seed to assign stable experiment buckets in DuckDB. Explore rollout thresholds and add users without renumbering everyone.
-
Hashfuncs for DuckDB 2.0: row hashing, cross-system compatibility, and Iceberg buckets
Hashfuncs now hashes whole rows, matches Spark and BigQuery fingerprints, reproduces Kafka and Iceberg buckets, and adds Jump consistent hashing for growing shard counts.