Why Parquet, and Why It Works So Well on S3

If you’ve been following the OLAP series, Parquet will look familiar — it’s essentially the production-grade, standardized version of what we built in Phase 1 and Phase 2. Phase 1’s columnar vectors map to Parquet’s column-by-column layout within row groups. Phase 2’s row groups with column segments and zone maps map directly to Parquet’s row groups with column chunks and min/max statistics. The main structural difference is that our format stores segment headers inline before each column’s data, so you walk through the file sequentially. Parquet moved all that metadata into a footer with precomputed absolute byte offsets — and that single design decision is what makes the S3 range-request pattern possible. This post is about why that matters.

Parquet is a columnar binary file format with a metadata footer at the end of the file. The footer contains the schema, the location (byte offset and length) of every column chunk in every row group, and statistics like min/max and null counts for each. Because all of this is laid out with precise byte offsets, a reader that can fetch arbitrary byte ranges can navigate a Parquet file surgically — read the footer, decide what it actually needs, and fetch only those bytes.

Why It’s Good on Its Own

• Columnar layout: data is stored column by column within row groups (typically 128 MB–1 GB). Analytical queries usually touch a few columns out of many, so you read a small fraction of the bytes.
• Per-column compression and encoding: dictionary, run-length, delta, bit-packing chosen per column, then Snappy/Zstd/Gzip on top. Typical 5–10x compression, sometimes much more. Columnar data compresses better than rows because adjacent values are similar.
• Statistics for pruning: min/max/null counts per row group, optional page index, optional bloom filters. The engine skips entire row groups based on WHERE clauses without reading them.
• Self-describing: schema and types live in the file. No external metadata needed.
• Native nested types: structs, lists, maps via Dremel-style repetition/definition levels.
• Open spec, broad implementations: C++, Java, Rust, Go. Every analytical tool reads and writes it.

Why It’s Specifically Great on S3

This is the part that turned Parquet from “a good format” into “the format.” The footer layout combined with HTTP range requests means you can run analytical queries directly against files in object storage without downloading them.

The read pattern:

1. HEAD request to get file size.
2. Range request for the tail of the file → parse the footer.
3. Use the footer’s statistics to eliminate row groups; use its byte offsets to locate the surviving column chunks for the columns you actually need.
4. Range requests for just those byte ranges.

For SELECT count(*), step 4 is skipped entirely — row counts are in the footer.

This depends on three things stacking up: HTTP supports range requests (RFC 2616, 1999); S3 implements them faithfully (since launch in 2006); Parquet puts metadata with absolute byte offsets in its footer. Pull any one of these out and the lakehouse architecture collapses — every query would have to download whole files. The byte-offset map is what makes the columnar layout actually pay off remotely. CSV in S3 is just files. Parquet in S3 is a queryable database substrate.

Why This Matters for Lakehouses

Iceberg, Delta Lake, and DuckLake all chose Parquet as their data layer. They add a layer above — manifests that track which files make up a table, aggregated file-level statistics, transactional commit logs — but they’re piggybacking on Parquet’s per-file pruning. This gives two layers of skip-the-work:

• Manifest-level: “of 10,000 files, only 23 could match this filter.”
• Footer-level: “in this file, only 2 of 40 row groups could match; here are the exact byte ranges for the 3 columns you asked for.”

Both layers depend on byte offsets and statistics being part of the format itself.

How It Compares to Other Formats

CSV / JSON / JSONL — row-oriented text, no types, no compression, no statistics, no random access. Fine for interchange and small data, hopeless for analytical scale. Every byte read every time.

Avro — row-oriented binary with embedded schema. Compact, strong schema evolution, good for write-heavy and streaming workloads (Kafka, Iceberg/Delta manifests). Can’t skip columns because rows interleave them, no useful pruning statistics. The rule of thumb: Avro for write/transport, Parquet for read/analytics.

ORC — Parquet’s direct columnar competitor, technically very similar and arguably more polished on some axes (stripe statistics, predicate pushdown). Lost the ecosystem war: stayed in the Hive/Hortonworks world while Parquet got picked up by Spark, AWS, and the broader cloud ecosystem.