Skip to content

DuckDB Extension

Hashfuncs

Compute non-cryptographic hashes in DuckDB with xxHash, MurmurHash3, RapidHash, wyhash, komihash, SipHash, CRC-32C and FNV-1a, and reproduce BigQuery, Spark, Kafka, Iceberg and ClickHouse hashes. Use them for partitioning, row keys, cache keys, and filter inputs.

10,456,551
extension loads ยท last 90 days
On this page
Sample data for the examples

Each browser example starts with these sample tables; sample files are shared across examples on this page. Run this setup first when trying the examples in your own DuckDB session.

-- Fixtures for the hashfuncs cookbook / technical-details examples.
--
-- The hashfuncs extension itself is pure CPU (no catalog, no network),
-- so these fixtures only exist to satisfy the *table references* the
-- documentation makes ("FROM events", "FROM users", ...). The numbers
-- chosen are tiny and deterministic so re-running the suite is stable.
--
-- Idempotent: every CREATE uses OR REPLACE.

-- The bitfilters cookbook section pairs hashfuncs with the bitfilters
-- extension. Load it best-effort so xor8_filter / xor8_filter_contains
-- resolve at parse time. If the community binary isn't available on the
-- current platform we still want the rest of the fixtures to apply.
INSTALL bitfilters FROM query_farm;
LOAD bitfilters FROM query_farm;

-- cookbook.mdx โ€” "Hash partitioning" and "Independent hashes via seeds"
CREATE OR REPLACE TABLE users AS
SELECT i AS user_id
FROM range(0, 8) t(i);

-- cookbook.mdx โ€” "Cross-system routing (Cassandra / Spark)"
CREATE OR REPLACE TABLE rows_to_route AS
SELECT 'k' || i::VARCHAR AS key
FROM range(0, 4) t(i);

-- cookbook.mdx โ€” "String cache keys"
CREATE OR REPLACE TABLE query_log AS
SELECT i AS user_id,
       'select ' || i::VARCHAR AS query_text
FROM range(0, 4) t(i);

-- cookbook.mdx โ€” "Bloom / XOR / Cuckoo filter input"
CREATE OR REPLACE TABLE allowed_users AS
SELECT 'user' || i::VARCHAR || '@example.com' AS email
FROM range(0, 4) t(i);

-- cookbook.mdx โ€” second bitfilters block reads from `filter_table f`
-- (a one-row table holding a pre-built xor8 filter).
CREATE OR REPLACE TABLE filter_table AS
SELECT xor8_filter(xxh3_64(email)) AS filter
FROM allowed_users;

-- cookbook.mdx โ€” "Independent hashes via seeds" uses `FROM keys`
CREATE OR REPLACE TABLE keys AS
SELECT 'k' || i::VARCHAR AS key
FROM range(0, 4) t(i);

-- cookbook.mdx + technical-details.mdx โ€” "events" with event_id, payload
-- Used by sharding / dedup / "magic moment" examples.
CREATE OR REPLACE TABLE events AS
SELECT i AS event_id,
       'evt-' || i::VARCHAR AS payload,
       'user' || i::VARCHAR || '@example.com' AS email
FROM range(0, 8) t(i);

-- technical-details.mdx โ€” "double-hash with two seeds" reads `FROM data`
CREATE OR REPLACE TABLE data AS
SELECT 'k' || i::VARCHAR AS key
FROM range(0, 4) t(i);

Create repeatable hash values for data processing

Hashfuncs exposes named non-cryptographic algorithms such as xxHash, MurmurHash, and RapidHash. They turn a value into a fixed-size number that can be used to assign partitions, construct lookup keys, or compare fingerprints across systems.

Choose a specific algorithm, seed, and input representation when another program must reproduce the result. Hash collisions are possible, so an equal hash alone does not establish that two records are equal. For cryptographic digests or HMACs, use Crypto.

Example

Assign keys to hash partitions

Hash each sample key with the same seed and map it to one of four partition numbers.

Tested with DuckDB 1.5.6. Browser examples use Haybarn-Wasm 2.0.0-rc12.

SET allow_extension_repositories = 'allowed';
CREATE EXTENSION REPOSITORY IF NOT EXISTS query_farm
  WITH PREFIX 'https://duckdb-extensions.query.farm';
INSTALL hashfuncs FROM query_farm;
LOAD hashfuncs FROM query_farm;

WITH users(user_id) AS (VALUES ('alice'), ('bob'), ('carol'))
SELECT user_id, xxh3_64(user_id, 42) % 4 AS partition_id
FROM users
ORDER BY user_id;

Expected result

user_id partition_id
alice 2
bob 2
carol 3

Keep the algorithm, input type, and seed consistent wherever you compute the partition. These hashes are not suitable for passwords or message authentication.

Common tasks

Deep Dive

Technical Details

Reference

Extension Contents

Quick reference to all available functions and settings organized by category.

CRC-32C & FNV-1a
crc32c() Object type: Scalar function CRC-32C (Castagnoli) checksum as UINTEGER โ€” the checksum used by iSCSI, ext4, Google Cloud Storage objects and many storage and wire formats.
fnv1a_32() Object type: Scalar function 32-bit FNV-1a.
fnv1a_64() Object type: Scalar function 64-bit FNV-1a.
Consistent hashing
jump_hash() Object type: Scalar function Jump consistent hash (Lamping and Veach): assigns a 64-bit key to one of buckets buckets so that growing from n to n + 1 buckets moves only about 1/(n + 1) of the keys โ€” all of them into the new bucket โ€” where hash % n would move almost every key.
Matching other systems
clickhouse_cityhash64() Object type: Scalar function ClickHouse's cityHash64 of a String: CityHash64 from CityHash v1.0.2, the version ClickHouse ships, as UBIGINT.
farm_fingerprint() Object type: Scalar function FarmHash Fingerprint64 as a signed BIGINT โ€” the same value as BigQuery's FARM_FINGERPRINT for the same STRING or BYTES.
iceberg_bucket() Object type: Scalar function The partition Apache Iceberg's bucket[N] transform assigns a value: (murmur3_x86_32(value) & Integer.MAX_VALUE) % N, using the Iceberg spec's encoding of each type.
iceberg_hash() Object type: Scalar function Iceberg's 32-bit bucket hash (spec Appendix B) as a signed INTEGER: MurmurHash3 x86_32 with seed 0 over Iceberg's encoding of the value โ€” integers as 8-byte little-endian longs, DECIMAL as the minimal big-endian bytes of its unscaled value, dates as days, times and timestamps as microseconds (nanosecond timestamps floored), strings as UTF-8 and UUIDs in big-endian order.
kafka_murmur2() Object type: Scalar function Kafka's murmur2 hash of a serialized record key (VARCHAR as UTF-8 bytes, or BLOB), as a signed INTEGER โ€” the hash Kafka's default partitioner applies to keyed records.
kafka_partition() Object type: Scalar function The partition Kafka's default partitioner assigns a keyed record: toPositive(murmur2(key)) % partitions.
spark_hash() Object type: Scalar function Spark's hash(): Murmur3 x86_32 with seed 42, returning a signed INTEGER.
spark_xxhash64() Object type: Scalar function Spark's xxhash64(): XXH64 with seed 42, returning a signed BIGINT, with the same multi-value seeding, NULL skipping and type rules as spark_hash.
MurmurHash3
murmurhash3_128() Object type: Scalar function 128-bit MurmurHash3 โ€” the x86 variant.
murmurhash3_32() Object type: Scalar function 32-bit MurmurHash3.
murmurhash3_x64_128() Object type: Scalar function 128-bit MurmurHash3 โ€” the x64 variant.
RapidHash
rapidhash() Object type: Scalar function RapidHash โ€” a 64-bit non-cryptographic hash designed for exceptional speed while keeping competitive distribution quality.
rapidhash_micro() Object type: Scalar function RapidHash Micro โ€” the upstream compact RapidHash variant with a smaller code footprint.
rapidhash_nano() Object type: Scalar function RapidHash Nano โ€” the smallest upstream RapidHash variant.
SipHash
siphash13() Object type: Scalar function SipHash-1-3 โ€” the faster, reduced-round SipHash used by Rust's standard library and CPython 3.11+ for string hashing.
siphash24() Object type: Scalar function SipHash-2-4, the standard SipHash, returning the 64-bit result as UBIGINT.
wyhash & komihash
komihash() Object type: Scalar function 64-bit komihash, version 5 โ€” a fast, high-quality hash whose output has been stable since 5.0 and matches hash4j's komihash5_0.
wyhash() Object type: Scalar function 64-bit wyhash, final version 4 with its original default secret โ€” the version implemented by Java's hash4j (wyhashFinal4).
xxHash
xxh3_128() Object type: Scalar function 128-bit xxHash3 (XXH3_128) returned as a UHUGEINT.
xxh3_128_hex() Object type: Scalar function 128-bit xxHash3 returned as a 32-character lowercase hex string in canonical xxHash byte order (high64 || low64), as defined by the xxHash specification.
xxh3_64() Object type: Scalar function 64-bit xxHash3 (XXH3_64) โ€” the modern xxHash variant tuned for vectorized CPUs (SSE2/AVX2).
xxh32() Object type: Scalar function 32-bit xxHash (XXH32) โ€” the classic 32-bit member of the xxHash family.
xxh64() Object type: Scalar function 64-bit xxHash (XXH64).

API Reference

Function Reference

Practical Examples

Cookbook

Real-world recipes and patterns for common use cases.

Platform Support

Compatibility

Extension availability may vary by platform and DuckDB version. Check below to ensure this extension supports your environment before installation.

Quick Facts

Release status Stable
Software License MIT
Pricing Free
Written In C++
Source Available Yes
View on GitHub
Usage
10,456,551
loads ยท last 90 days

Platforms

DuckDB 2.0.0

  • Linux x86_64aarch64
  • Linux (musl) Not available
  • macOS Apple Silicon
  • Windows x86_64x86_64 (MinGW)
  • WASM Not available
Compiled binary sizes
Platform Architecture Size
Linux x86_64 4.00 MB
Linux aarch64 3.36 MB
macOS Apple Silicon 2.83 MB
Windows x86_64 10.23 MB
Windows x86_64 (MinGW) 17.85 MB

Compressed download sizes for the release shown above.

DuckDB & Haybarn

Release calendar
  • DuckDB v2.0.0 Query.Farm repository Supported
  • Haybarn v1.5.6 Built-in community repository Supported

Browser examples use Haybarn-Wasm 2.0.0-rc12.

Checked .

Learn and explore

Tutorials and articles

Writing about Hashfuncs: how it works, what changed, and why.