DuckDB VARIANT and Arrow: What Works Today

DuckDB 2.0 makes VARIANT a first-class semi-structured type. DuckDB can discover common structure, shred it into physical columns, push extraction into scans, and read or write Parquet Variant. That pipeline is described in the DuckDB 2.0 preview.

Arrow interoperability is a separate piece of work. There is now a standard representation—arrow.parquet.variant—but each Arrow implementation still has to recognize the extension, validate its storage, and expose useful APIs for its language.

The badges below distinguish released code from work that is still under review.

The DuckDB side

Direct Arrow import and export remain the main gap, tracked in duckdb/duckdb#24091. On the current DuckDB 2.0 branch, exporting SELECT 42::VARIANT to Arrow raises Unsupported Arrow type VARIANT. Importing a canonical Variant column loses the logical type and produces its bare STRUCT(metadata BLOB, value BLOB) storage instead.

PR #24157 spans nine commits, 18 changed files, and more than 700 lines of Arrow tests. It registers arrow.parquet.variant, converts through DuckDB’s Parquet Variant encoder and decoder, preserves SQL nulls, resolves fields by name, and tests Binary, LargeBinary, BinaryView, dictionary-encoded metadata, and run-end-encoded metadata.

Its CI is green and it has an approval, but a DuckDB maintainer still has changes requested. The remaining review covers code placement, large nested-vector capacity, binary child validation, and a few conversion cleanups. The head commit is 83cbc26770; none of those commits is in the DuckDB 2.0 release branch yet.

Support by Arrow implementation

Arrow defines Variant as a Struct containing non-null metadata and either value, typed_value, or both. A library can preserve that Struct without understanding the value inside it. The more useful question is whether it can encode, decode, shred, or inspect Variant data.

C++

◐Partial

Arrow C++ already defines the canonical extension type and maps it to Parquet schemas. Reading, writing, validation, shredding, and unshredding are still under review in PR #50252.

If DuckDB PR #24157 lands as written, C++ callers will be able to exchange unshredded values through the Arrow C Data interface. Native C++ value handling will still depend on the Arrow work.

Python / PyArrow

↻Waiting

PyArrow can preserve the underlying Struct and its field metadata, but it has no public VariantType, VariantArray, or VariantScalar.

After the DuckDB PR, a carefully constructed canonical schema should enter DuckDB as VARIANT. Python still will not have a natural way to inspect those values until #50131 and #50132 land.

Go

✓Available

Go currently has the most complete released implementation. Arrow Go PR #434 shipped in v18.4.0 with pqarrow round trips for both unshredded and shredded Variant.

Canonical unshredded values should round-trip directly once the DuckDB PR lands. Shredded values must first be unshredded before DuckDB can accept them.

Rust

◐Experimental

arrow-rs includes Variant arrays, builders, JSON conversion, path kernels, shredding, and unshredding behind its variant_experimental feature.

That provides broad coverage for unshredded exchange, although the APIs remain unstable and DuckDB will reject shredded input. Stabilization is tracked in #10546.

Java

◐Partial

Arrow Java 19 added an unshredded VariantVector plus readers and writers.

Java currently registers the older parquet.variant extension name rather than the canonical arrow.parquet.variant. Direct exchange therefore needs an adapter for the name and metadata.

Other Arrow implementations

→Raw transport

An implementation that preserves the Struct and its field metadata can carry the bytes without offering a semantic Variant API. After the DuckDB PR, it should be able to relay DuckDB’s unshredded representation, but applications must decode the binary format themselves to inspect values.

Already available

• The Parquet Variant encoding and shredding specifications are finalized.
• arrow.parquet.variant is an official Arrow canonical extension type.
• Arrow C++ has its VariantExtensionType and the corrected canonical extension name.
• Arrow Go has released full Parquet/Arrow Variant round trips.
• Arrow Java has released its unshredded Variant module, and Rust releases contain the experimental Variant crates and kernels.
• DuckDB 2.0 has the database and Parquet side: storage, shredding, execution, functions, and Parquet import/export.

Still in flight

• DuckDB: merge PR #24157, then decide how and when to support Arrow typed_value shredding.
• Arrow C++: merge PR #50252 for full read, write, validation, shredding, and unshredding.
• PyArrow: expose Variant types and arrays in #50131, followed by Parquet integration in #50132.
• Rust: close the stabilization list and replace variant_experimental with a stable feature.
• Java: align parquet.variant with the canonical arrow.parquet.variant name and add shredded support.

The practical fallback

For portable application output, cast a DuckDB Variant to JSON before crossing the Arrow boundary:

SELECT payload::JSON AS payload
FROM events;

That gives up native Variant typing, but it works across today’s Arrow clients and makes the compromise visible in the query.

DuckDB’s Parquet extension also exposes a lower-level bridge: variant_to_parquet_variant() produces the canonical metadata and value blobs, while variant_bytes_to_variant(metadata || value) reconstructs the DuckDB value. That path can preserve the binary representation when both ends are under your control, but it does not turn the Arrow column into the canonical extension automatically and should be treated as a version-sensitive integration technique.

This is a snapshot as of 13 September 2026. We may revisit it as the work moves, particularly after DuckDB 2.0 ships.