Use parquet_metadata() to Understand Parquet Filter Performance

Same rows. Same filter. Different opportunities to skip work. A Parquet file can keep each day’s events together or scatter every day across every row group. The query’s answer stays the same, but the metadata describes very different ranges.

We will write both layouts, inspect them with parquet_metadata(), and identify which row groups could contain one selected day. This is a way to investigate a slow filter before guessing at compression or adding hardware.

One dataset, two real files

There are 8,192 events over four days, with 2,048 events per day. The first file sorts by date. The second interleaves the four dates. Both use a requested row-group size of 2,048; we verify the resulting groups rather than assuming the writer produced them.

Download the complete sorted Parquet file and mixed Parquet file. The browser creates its own copies from the same data. File paths come from getvariable('sorted_file') and getvariable('mixed_file'); locally, you can use 'sorted.parquet' and 'mixed.parquet' in their place.

Read metadata for the column you filter

parquet_metadata() exposes information for column chunks within row groups. It does not return one row per event or simply one row per row group. Filter path_in_schema to the column whose statistics you want to inspect.

SELECT row_group_id, row_group_num_rows AS rows,
       stats_min_value AS first_day, stats_max_value AS last_day
FROM parquet_metadata(getvariable('sorted_file'))
WHERE path_in_schema = 'event_day' ORDER BY row_group_id;

Each group contains 2,048 rows, and each group’s date minimum equals its maximum. Group 1 holds January 2. The other groups’ ranges cannot contain that date.

row_group_id is local to a file. When querying a glob of files, keep file_name with the group ID so group 1 in one file is not confused with group 1 in another. Also avoid summing row_group_num_rows across all column-chunk records: that repeats the same group’s row count for each column.

Compare the mixed layout

Now inspect the same column in the interleaved file:

SELECT row_group_id, row_group_num_rows AS rows,
       stats_min_value AS first_day, stats_max_value AS last_day
FROM parquet_metadata(getvariable('mixed_file'))
WHERE path_in_schema = 'event_day' ORDER BY row_group_id;

Every group ranges from January 1 to January 4. A filter for January 2 overlaps every range. Min/max statistics cannot rule out any of these groups, even though only a quarter of the records match.

A row group’s bounds describe possibilities, not its complete contents. If a group’s range is January 1–4, it need not contain every date between them. Overlap means “might match”; a disjoint range can prove that it cannot match, under the relevant type and comparison semantics.

Keep the answer separate from the work

Run the same date filter against both files:

SELECT 'sorted' AS layout, count(*) AS matching_rows
FROM read_parquet(getvariable('sorted_file')) WHERE event_day = DATE '2026-01-02'
UNION ALL
SELECT 'mixed', count(*)
FROM read_parquet(getvariable('mixed_file')) WHERE event_day = DATE '2026-01-02';

Both return 2,048 matching rows. This checks the logical answer. It does not prove how many bytes were read or how much time the query saved.

DuckDB can push supported filters into a Parquet scan and use statistics to avoid unnecessary work. Inspect the actual plan with EXPLAIN and the measured execution with EXPLAIN ANALYZE. A filter that is not available to the scan cannot benefit in the same way from these ranges.

Identify candidate groups yourself

The metadata’s statistics fields are strings. Cast them to the column’s type before comparing them. A missing or unparseable bound cannot safely prove exclusion, so the challenge keeps those groups as candidates.

SELECT row_group_id
FROM parquet_metadata(getvariable('sorted_file'))
WHERE path_in_schema = 'event_day'
  AND (TRY_CAST(stats_min_value AS DATE) IS NULL
       OR TRY_CAST(stats_min_value AS DATE) <= DATE '2026-01-03')
  AND (TRY_CAST(stats_max_value AS DATE) IS NULL
       OR TRY_CAST(stats_max_value AS DATE) >= DATE '2026-01-03')
ORDER BY row_group_id;

This is a diagnostic query for our DATE fixture, not a replacement for the scanner’s pruning logic. For other types, consider ordering, truncated or inexact statistics, and fields such as min_is_exact and max_is_exact. NULL predicates also need their own reasoning, including stats_null_count.

Use the finding to choose an experiment

If many groups span the full range of a frequently filtered column, try a layout that clusters related values together and measure the same query again. Sorting for one access pattern can help that pattern while doing less for another. Writing more, smaller groups also has metadata and processing costs; 2,048 is deliberately small for this demonstration, not a production recommendation.

For size investigations, total_compressed_size describes an individual column chunk. It is useful for comparing stored layouts, but summing candidate chunks is not a measurement of actual query I/O: metadata, caching, page reads, and other behavior also matter.

Start with the range evidence, choose one layout change, verify equivalent results, and measure on representative data. The file can only skip work that its layout and the query make possible.