Blog - kimserey lam

Oct 28, 2026

DuckDB Memory Management

DuckDB defaults to consuming 80% of system RAM for its buffer pool. On a developer laptop with 16 GB, that’s fine. In a 2 GB Kubernetes pod, that’s 1.6 GB — leaving 400 MB for the Go or JVM runtime, the application, and the OS. That’s how you get OOM-killed. This post walks through how DuckDB manages memory in-process, what happens when queries exceed the memory limit, and how to tune settings for container deployments.

Oct 24, 2026

Lakehouse – Phase 8 Storage and Access

Up to this point, the lake engine reads and writes Parquet files directly on the local filesystem. This works for development, but a real lakehouse stores data on object storage (S3, GCS, Azure Blob) — that is the whole point of separating compute from storage. This final phase introduces a Storage interface that abstracts blob I/O, adds an S3 implementation alongside the local backend, and provides a CLI REPL for interacting with the lake.

Oct 21, 2026

Lakehouse – Phase 7 Deletes and Updates

Parquet files are immutable. Once written, you cannot remove row 42 without rewriting the entire file. But lakehouse tables need DELETE and UPDATE. The solution is positional delete files: small secondary files that record which rows in a data file have been logically removed. During scans, the reader loads delete positions and skips those rows. UPDATE decomposes into DELETE + INSERT within a single transaction.

Oct 17, 2026

Lakehouse – Phase 6 Schema Evolution

In a traditional database, changing a column’s name is fast because the engine controls the storage format. In a lakehouse, data lives in immutable Parquet files. You cannot edit a Parquet file — it is a sealed, self-describing unit. Rewriting petabytes of files just to add a column would take hours. Schema evolution solves this: change the schema in metadata without touching the data files. Old files still have the old column set; new files have the new columns. The reader reconciles the difference using field IDs — immutable integer identifiers assigned to each column at creation time.

Oct 14, 2026

Lakehouse – Phase 5 Snapshots and Time Travel

Every mutation to a lakehouse catalog — INSERT, DELETE, ALTER TABLE — produces a new snapshot. A snapshot is a monotonically increasing integer ID that represents the state of all tables, columns, and data files at that instant. Nothing is overwritten; metadata rows carry visibility ranges that say when each row was “born” and when it was “retired.” This is the foundation of time travel: query any historical state by passing a different snapshot ID to the same visibility filter.

Oct 10, 2026

Lakehouse – Phase 4 Scanning and Query

Phase 3 showed how data enters the lakehouse. This phase shows how data comes back out. A SELECT query goes through a pipeline with a critical optimization in the middle: statistics-based file pruning. The scan engine never blindly reads all data files. It first queries the metadata catalog to find which files belong to the table at the current snapshot, compares the WHERE clause against each file’s column statistics, and skips files that cannot possibly contain matching rows. This is where the database-backed catalog shines brightest — file pruning is a SQL query.

Oct 07, 2026

Lakehouse – Phase 3 Ingest Pipeline

Phase 1 gave us a metadata catalog that can track tables, columns, files, and statistics. Phase 2 gave us the ability to write and read Parquet files with column statistics. This phase connects them: an INSERT writes data to a Parquet file and then atomically registers that file (with its statistics) in the metadata catalog. The design has one core principle: data files are written before metadata is updated, and the metadata update is atomic.

Oct 03, 2026

Lakehouse – Phase 2 Parquet Data Files

Phase 1 built the metadata catalog — Postgres tables that track what tables exist, what columns they have, and which files belong to them. But metadata alone is useless without actual data. This phase builds the data file layer: writing and reading Apache Parquet files with per-column statistics that the metadata catalog will reference for file pruning. We use the parquet-go library rather than implementing Parquet from scratch — the educational value is in understanding how Parquet’s structure enables file pruning and schema evolution.

Sep 30, 2026

Lakehouse – Phase 1 Metadata Catalog

Every lakehouse has two layers: data files (Parquet files containing actual rows) and metadata (which files exist, what columns they have, what value ranges each file contains). The metadata layer is what separates a lakehouse from a pile of Parquet files. Apache Iceberg stores this metadata as a tree of manifest files (JSON/Avro) on object storage. DuckLake takes a different approach: it stores metadata in a relational database (Postgres, MySQL, or SQLite). This is not just a different storage location — it fundamentally changes how the entire system works.

Sep 26, 2026

AWS Credential Chain in DuckDB

When DuckDB reads or writes Parquet files in S3, it needs AWS credentials. Rather than hardcoding access keys, you can tell DuckDB to discover credentials at runtime using the same chain the AWS SDK uses — environment variables, IRSA tokens, SSO profiles, ECS task roles, or EC2 instance metadata. This post covers how the credential chain works, what each step looks for, and how it connects to DuckLake in both local development and production.

Sep 23, 2026

Deploying DuckLake on AWS with CloudFormation

DuckLake needs two pieces of infrastructure: an S3 bucket for Parquet data files and a PostgreSQL database for the metadata catalog. This post walks through standing up that infrastructure on AWS using CloudFormation — a single YAML template that creates all the resources as a unit. Along the way, we cover KMS encryption, bucket policies, Aurora PostgreSQL configuration, IAM roles for workload access, and the deployment workflow.

Sep 19, 2026

OpenTelemetry Delta Temporality and Datadog

When you ship OpenTelemetry metrics to Datadog, the choice between cumulative and delta temporality changes how your dashboards behave. This post covers the practical differences, what Datadog actually stores, and how to query each instrument type correctly.

Sep 16, 2026

OpenTelemetry in Go — Traces, Metrics, Logs and the Collector

OpenTelemetry gives you traces, metrics, and logs from a single instrumentation library with a vendor-neutral export path. This post walks through adding OpenTelemetry to a Go HTTP server from scratch — setting up providers, auto-instrumenting HTTP, creating manual spans, recording custom metrics, emitting correlated logs, configuring the Collector, and running the full stack with Docker Compose. It finishes with a side-by-side comparison of the OTel metrics API against the Prometheus client library.

Sep 12, 2026

Go Building Services — HTTP, Signals, Migrations and Reflection

The previous tutorials covered Go’s type system, concurrency primitives, and error handling. This one shifts to production patterns you need when building real services: serving HTTP requests, managing process lifecycle with signal handling, running database migrations with embedded files, and using reflection for metaprogramming. Each topic stands on its own, but together they represent the kind of infrastructure code that appears in nearly every Go service.

Sep 09, 2026

Go Standard Library — Packages, IO, JSON and Testing

This post covers four day-to-day essentials for writing Go programs: organizing code into packages and modules, streaming data through the io.Reader and io.Writer interfaces, serializing and deserializing JSON, and writing tests. These are the building blocks you will reach for in nearly every Go project — packages keep code organized, the I/O interfaces let you compose data pipelines, JSON handles serialization, and the testing package gives you a lightweight framework for verifying everything works.