Aug 22, 2026

DuckLake — Open Lakehouse with DuckDB

DuckLake is a lakehouse format built as a DuckDB extension. It separates metadata from data: a catalog (a local .ducklake file or a PostgreSQL database) tracks table schemas, snapshots, and file pointers, while Parquet files hold the actual data on a local filesystem or in S3-compatible object storage. This separation is the central design decision — the catalog knows what files exist, what columns they contain, and what value ranges they hold, but it never stores the data itself. In this post, we’ll go from running DuckLake locally to running it with PostgreSQL and MinIO, look at how partitioning organizes data in S3, and see how DuckLake’s query engine prunes files and pushes predicates down to avoid reading data it doesn’t need.

What Is DuckLake

Traditional databases store metadata and data together. Lakehouse formats like Iceberg, Delta Lake, and Hudi split them apart — they use a metadata layer (manifests, commit logs) on top of Parquet files in object storage. DuckLake follows the same philosophy but uses a relational database as the catalog instead of JSON/Avro manifest files.

The catalog stores:

Table definitions — schemas, column types, partition keys.
Data file registry — which Parquet files belong to which table, their row counts, and file sizes.
Column statistics — min/max values per column per file, used for pruning at query time.
Snapshots — a complete version history of every change, enabling time travel.

The data files are Parquet, stored wherever you point DATA_PATH — a local directory, an S3 bucket, or any S3-compatible object store like MinIO or SeaweedFS.

Because the catalog is a regular database, you get transactions, concurrent access (with PostgreSQL), and the ability to query metadata with SQL. And because the data files are plain Parquet, any tool that reads Parquet can access them independently of DuckLake.

Running Locally

The simplest setup uses a local .ducklake file for metadata and a local directory for data. No Docker, no object storage, no external dependencies.

Install and load the extension:

INSTALL ducklake;
LOAD ducklake;

DuckLake inlines small tables (10 rows or fewer) directly into the catalog to avoid creating tiny Parquet files. For testing, disable this so all data goes to Parquet:

SET ducklake_default_data_inlining_row_limit = 0;

Attach a DuckLake catalog. The first argument is the metadata path (prefixed with ducklake:), and DATA_PATH is where Parquet files will be written:

ATTACH 'ducklake:step1_metadata.ducklake' AS lake (DATA_PATH 'step1_data/');
USE lake;

Create a table and insert data:

CREATE OR REPLACE TABLE sensors (
    tenant_id   VARCHAR,
    sensor_id   INTEGER,
    ts          TIMESTAMP,
    reading     DOUBLE,
    location    VARCHAR
);

INSERT INTO sensors
SELECT
    CASE WHEN i % 2 = 0 THEN 'tenant_a' ELSE 'tenant_b' END,
    i,
    TIMESTAMP '2025-01-01' + INTERVAL (i) MINUTE,
    round(random() * 100, 2),
    CASE WHEN i % 3 = 0 THEN 'NYC' WHEN i % 3 = 1 THEN 'SF' ELSE 'CHI' END
FROM range(10000) t(i);

After the insert, step1_data/ contains the Parquet files:

step1_data/
  main/sensors/
    ducklake-<uuid>.parquet

And step1_metadata.ducklake is a DuckDB file that holds the catalog tables — table definitions, column stats, snapshot history, and pointers to the Parquet files.

You can inspect the catalog with built-in functions:

SELECT * FROM ducklake_snapshots('lake');
SELECT * FROM ducklake_table_info('lake');

This setup works for local development and single-user workflows. But it has a limitation: the .ducklake file uses an exclusive file lock. If two processes try to write at the same time, the second one gets an error immediately:

Could not set lock on file "step1_metadata.ducklake":
Conflicting lock is held in duckdb (PID 46829)

For concurrent writes, you need PostgreSQL as the catalog backend.

Running with PostgreSQL and MinIO

A production-like setup replaces the local .ducklake file with PostgreSQL (for concurrent metadata access) and the local directory with MinIO (an S3-compatible object store for data).

A docker compose file brings both services up:

services:
  minio:
    image: quay.io/minio/minio:latest
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio_data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 5s
      timeout: 5s
      retries: 5

  postgres:
    image: postgres:16
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: ducklake
      POSTGRES_PASSWORD: ducklake
      POSTGRES_DB: ducklake
    volumes:
      - pg_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ducklake"]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  minio_data:
  pg_data:

Start the services and create a bucket for DuckLake data:

docker compose up -d
docker compose exec minio mc mb local/ducklake-data

Now load the required extensions — ducklake for the lakehouse logic, httpfs for S3 access, and postgres for the PostgreSQL catalog backend:

INSTALL ducklake; LOAD ducklake;
INSTALL httpfs;   LOAD httpfs;
INSTALL postgres; LOAD postgres;

SET ducklake_default_data_inlining_row_limit = 0;

Create an S3 secret pointing at MinIO:

CREATE SECRET (
    TYPE s3,
    KEY_ID 'minioadmin',
    SECRET 'minioadmin',
    ENDPOINT 'localhost:9000',
    USE_SSL false,
    URL_STYLE 'path'
);

Attach the DuckLake catalog using PostgreSQL for metadata and MinIO for data:

ATTACH 'ducklake:postgres:dbname=ducklake user=ducklake password=ducklake host=localhost port=5432'
    AS lake (DATA_PATH 's3://ducklake-data/');
USE lake;

From here, creating tables and inserting data is identical to the local setup. The difference is where things go: metadata rows land in PostgreSQL tables (ducklake_table, ducklake_column, ducklake_data_file, ducklake_snapshot), and Parquet files land in the MinIO bucket.

With PostgreSQL as the catalog, concurrent writes work. Two DuckDB processes can insert into the same table simultaneously — PostgreSQL handles the row-level locking internally. The data files in S3 are immutable (each insert creates new Parquet files), so there are no conflicts on the storage side either.

Partitioning

Partitioning tells DuckLake to organize Parquet files into a directory hierarchy based on column values. Instead of writing all rows into a single file, DuckLake creates separate files for each distinct combination of partition key values.

Define a table and set the partition key:

CREATE TABLE sensors (
    tenant_id   VARCHAR,
    sensor_id   INTEGER,
    event_date  DATE,
    ts          TIMESTAMP,
    reading     DOUBLE,
    location    VARCHAR
);

ALTER TABLE sensors SET PARTITIONED BY (tenant_id, event_date);

Insert 30,000 rows spanning 3 tenants and 10 dates:

INSERT INTO sensors
SELECT
    CASE WHEN i % 3 = 0 THEN 'tenant_a'
         WHEN i % 3 = 1 THEN 'tenant_b'
         ELSE 'tenant_c' END AS tenant_id,
    i AS sensor_id,
    DATE '2025-01-01' + INTERVAL (i % 10) DAY AS event_date,
    TIMESTAMP '2025-01-01' + INTERVAL (i) MINUTE AS ts,
    round(random() * 100, 2) AS reading,
    CASE WHEN i % 3 = 0 THEN 'NYC'
         WHEN i % 3 = 1 THEN 'SF'
         ELSE 'CHI' END AS location
FROM range(30000) t(i);

This produces 30 Parquet files — one for each (tenant, date) combination, each holding 1,000 rows.

How Data Is Organized in S3

The directory layout in S3 follows the Hive partitioning convention: column=value/ directories nested according to the partition key order.

For an unpartitioned table, all files sit in a flat directory:

s3://ducklake-data/
  main/sensors/
    ducklake-<uuid-1>.parquet
    ducklake-<uuid-2>.parquet

For a table partitioned by a single column (location):

s3://ducklake-data/
  main/sensors_by_location/
    location=CHI/
      ducklake-<uuid>.parquet
    location=NYC/
      ducklake-<uuid>.parquet
    location=SF/
      ducklake-<uuid>.parquet

For a table partitioned by two columns (tenant_id, event_date):

s3://ducklake-data/
  main/sensors/
    tenant_id=tenant_a/
      event_date=2025-01-01/
        ducklake-<uuid>.parquet
      event_date=2025-01-02/
        ducklake-<uuid>.parquet
      ...
    tenant_id=tenant_b/
      event_date=2025-01-01/
        ducklake-<uuid>.parquet
      ...
    tenant_id=tenant_c/
      ...

The path structure is {DATA_PATH}/{schema}/{table}/{partition_key=value}/.../{file}.parquet. Each Parquet file is a standard columnar file with row groups, column chunks, and a metadata footer containing min/max statistics and byte offsets.

The catalog in PostgreSQL (or the .ducklake file) tracks every one of these files: their paths, row counts, file sizes, and per-column statistics. DuckLake never needs to list the S3 bucket to find files — it queries the catalog instead. This is what makes pruning fast: the catalog is a regular database table, and filtering it is a local operation that doesn’t touch object storage at all.

Partition Pruning

Partition pruning is the first and most impactful optimization. When a query includes a WHERE clause on a partition column, DuckLake consults the catalog to determine which Parquet files could possibly match. Files that can’t match are eliminated before any S3 request is made.

Using the 30-file dataset (3 tenants x 10 dates), EXPLAIN ANALYZE shows exactly how many files DuckLake reads for each query:

Full scan (no filter) — 30 files:

EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors;
-- Total Files Read: 30

Both partition keys — 1 file:

EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE tenant_id = 'tenant_a' AND event_date = DATE '2025-01-03';
-- Total Files Read: 1

First partition key only — 10 files:

EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE tenant_id = 'tenant_a';
-- Total Files Read: 10

Second partition key only (no tenant filter) — 3 files:

EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE event_date = DATE '2025-01-03';
-- Total Files Read: 3

This last query is significant. Filtering on event_date without specifying tenant_id still prunes down to 3 files (one per tenant for that date). DuckLake prunes on the second partition key independently of the first. It doesn’t need the leading key to be present — it evaluates each partition column separately against the catalog.

Date range — 9 files:

EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE event_date BETWEEN DATE '2025-01-02' AND DATE '2025-01-04';
-- Total Files Read: 9

Range predicates prune the same way. Three dates across three tenants: 9 files.

The pruning stack has three layers, each narrowing the data further:

Partition pruning — skip entire Parquet files using partition column values in the catalog. No S3 requests.
Row group pruning — within a selected file, skip row groups using min/max stats from the Parquet footer.
Filter evaluation — scan the remaining rows and discard non-matches.

In EXPLAIN ANALYZE output, the key number to look for is Total Files Read in the TABLE_SCAN node:

┌───────────────────────────┐
│         TABLE_SCAN        │
│       Table: sensors      │
│          Filters:         │
│  event_date='2025-01-03'  │
│    Total Files Read: 3    │
│         3,000 rows        │
└───────────────────────────┘

Predicate Pushdown Through Views

Partition pruning only helps if the filter reaches the table scan. In real applications, queries often go through layers of views. Does DuckLake still prune when the filter is applied to a view several layers above the base table?

Stack three views on top of the partitioned table:

CREATE OR REPLACE VIEW v_base AS
SELECT tenant_id, event_date, sensor_id, reading, location FROM sensors;

CREATE OR REPLACE VIEW v_enriched AS
SELECT *, reading * 1.1 AS adjusted_reading FROM v_base;

CREATE OR REPLACE VIEW v_summary AS
SELECT
    v.tenant_id, v.event_date, v.adjusted_reading, v.location,
    AVG(v.adjusted_reading) OVER (PARTITION BY v.tenant_id) AS tenant_avg
FROM v_enriched v;

Query the top-level view with a date filter:

EXPLAIN ANALYZE
SELECT * FROM v_summary
WHERE event_date = DATE '2025-01-03';

The filter on event_date pushes all the way through v_summary → v_enriched → v_base → sensors and triggers partition pruning at the TABLE_SCAN node. Only 3 files are read, not 30. The views add zero overhead to the pruning — DuckDB’s optimizer traces the filter down to the underlying table before execution begins.

This also works through joins. A view that joins v_base to itself still benefits from pushdown:

CREATE OR REPLACE VIEW v_with_join AS
SELECT a.tenant_id, a.event_date, a.reading, b.reading AS other_reading
FROM v_base a
JOIN v_base b
  ON a.tenant_id = b.tenant_id
 AND a.event_date = b.event_date
 AND a.sensor_id != b.sensor_id;

EXPLAIN ANALYZE
SELECT * FROM v_with_join
WHERE event_date = DATE '2025-01-03'
LIMIT 10;