Blog - kimserey lam

Oct 10, 2026

Lakehouse – Phase 4 Scanning and Query

Phase 3 showed how data enters the lakehouse. This phase shows how data comes back out. A SELECT query goes through a pipeline with a critical optimization in the middle: statistics-based file pruning. The scan engine never blindly reads all data files. It first queries the metadata catalog to find which files belong to the table at the current snapshot, compares the WHERE clause against each file’s column statistics, and skips files that cannot possibly contain matching rows. This is where the database-backed catalog shines brightest — file pruning is a SQL query.

Oct 07, 2026

Lakehouse – Phase 3 Ingest Pipeline

Phase 1 gave us a metadata catalog that can track tables, columns, files, and statistics. Phase 2 gave us the ability to write and read Parquet files with column statistics. This phase connects them: an INSERT writes data to a Parquet file and then atomically registers that file (with its statistics) in the metadata catalog. The design has one core principle: data files are written before metadata is updated, and the metadata update is atomic.

Oct 03, 2026

Lakehouse – Phase 2 Parquet Data Files

Phase 1 built the metadata catalog — Postgres tables that track what tables exist, what columns they have, and which files belong to them. But metadata alone is useless without actual data. This phase builds the data file layer: writing and reading Apache Parquet files with per-column statistics that the metadata catalog will reference for file pruning. We use the parquet-go library rather than implementing Parquet from scratch — the educational value is in understanding how Parquet’s structure enables file pruning and schema evolution.

Sep 30, 2026

Lakehouse – Phase 1 Metadata Catalog

Every lakehouse has two layers: data files (Parquet files containing actual rows) and metadata (which files exist, what columns they have, what value ranges each file contains). The metadata layer is what separates a lakehouse from a pile of Parquet files. Apache Iceberg stores this metadata as a tree of manifest files (JSON/Avro) on object storage. DuckLake takes a different approach: it stores metadata in a relational database (Postgres, MySQL, or SQLite). This is not just a different storage location — it fundamentally changes how the entire system works.

Sep 26, 2026

AWS Credential Chain in DuckDB

When DuckDB reads or writes Parquet files in S3, it needs AWS credentials. Rather than hardcoding access keys, you can tell DuckDB to discover credentials at runtime using the same chain the AWS SDK uses — environment variables, IRSA tokens, SSO profiles, ECS task roles, or EC2 instance metadata. This post covers how the credential chain works, what each step looks for, and how it connects to DuckLake in both local development and production.

Sep 23, 2026

Deploying DuckLake on AWS with CloudFormation

DuckLake needs two pieces of infrastructure: an S3 bucket for Parquet data files and a PostgreSQL database for the metadata catalog. This post walks through standing up that infrastructure on AWS using CloudFormation — a single YAML template that creates all the resources as a unit. Along the way, we cover KMS encryption, bucket policies, Aurora PostgreSQL configuration, IAM roles for workload access, and the deployment workflow.

Sep 19, 2026

OpenTelemetry Delta Temporality and Datadog

When you ship OpenTelemetry metrics to Datadog, the choice between cumulative and delta temporality changes how your dashboards behave. This post covers the practical differences, what Datadog actually stores, and how to query each instrument type correctly.

Sep 16, 2026

OpenTelemetry in Go — Traces, Metrics, Logs and the Collector

OpenTelemetry gives you traces, metrics, and logs from a single instrumentation library with a vendor-neutral export path. This post walks through adding OpenTelemetry to a Go HTTP server from scratch — setting up providers, auto-instrumenting HTTP, creating manual spans, recording custom metrics, emitting correlated logs, configuring the Collector, and running the full stack with Docker Compose. It finishes with a side-by-side comparison of the OTel metrics API against the Prometheus client library.

Sep 12, 2026

Go Building Services — HTTP, Signals, Migrations and Reflection

The previous tutorials covered Go’s type system, concurrency primitives, and error handling. This one shifts to production patterns you need when building real services: serving HTTP requests, managing process lifecycle with signal handling, running database migrations with embedded files, and using reflection for metaprogramming. Each topic stands on its own, but together they represent the kind of infrastructure code that appears in nearly every Go service.

Sep 09, 2026

Go Standard Library — Packages, IO, JSON and Testing

This post covers four day-to-day essentials for writing Go programs: organizing code into packages and modules, streaming data through the io.Reader and io.Writer interfaces, serializing and deserializing JSON, and writing tests. These are the building blocks you will reach for in nearly every Go project — packages keep code organized, the I/O interfaces let you compose data pipelines, JSON handles serialization, and the testing package gives you a lightweight framework for verifying everything works.

Sep 05, 2026

Go Concurrency — Goroutines, Channels and Sync

Go was designed with concurrency as a first-class feature. Rather than relying on OS threads and shared-memory locking as the primary model, Go provides goroutines (lightweight threads managed by the Go runtime), channels (typed conduits for communication between goroutines), and a select statement for multiplexing channel operations. The standard library rounds this out with a context package for cancellation and deadlines, and a sync package for traditional mutual exclusion when channels are not the right fit. This tutorial walks through all five building blocks, starting from basic goroutines and working up to a concurrent cache protected by read-write locks and atomics.

Sep 02, 2026

Go Error Handling and Resource Management

Go treats errors as ordinary values — there are no exceptions, no try/catch, and no hidden control flow. Every function that can fail returns an error alongside its result, and the caller decides what to do with it. For cleanup, Go provides defer, which guarantees a function call runs when the enclosing function returns, regardless of how it returns. Together, explicit error returns and defer give you predictable error propagation and deterministic resource management without the complexity of exception hierarchies or finalizers. In this post we cover Go’s error handling patterns — from basic checks through wrapping, sentinel errors, and context-aware error classification — then move to defer patterns including LIFO ordering, loop pitfalls, closure capture semantics, named return manipulation, and panic recovery.

Aug 29, 2026

Go Type System — Interfaces, Embedding and Generics

Go takes a distinctive approach to polymorphism and code reuse. There are no classes, no inheritance hierarchies, and no explicit implements declarations. Instead, Go provides three orthogonal mechanisms: interfaces for abstraction, embedding for composition, and generics for type-safe parameterization. Each one is simple on its own, and together they cover the same ground that class hierarchies cover in other languages — with less coupling. In this post, we’ll work through all three, starting with how Go interfaces are satisfied implicitly, then moving to struct and interface embedding, and finishing with generic functions and types.

Aug 26, 2026

Go Basics — Types, Memory and Data Structures

This is the first tutorial in a series on Go. It covers three foundational topics: pointers and references (how Go passes data around and when mutations are visible), structs with methods and receivers (how Go models data and behavior without classes), and slices and maps (the two built-in data structures you will use constantly). Each section builds on the previous one — pointers explain why receiver types matter, and receiver types explain how slice and map patterns work in practice.

Aug 22, 2026

DuckLake — Open Lakehouse with DuckDB

DuckLake is a lakehouse format built as a DuckDB extension. It separates metadata from data: a catalog (a local .ducklake file or a PostgreSQL database) tracks table schemas, snapshots, and file pointers, while Parquet files hold the actual data on a local filesystem or in S3-compatible object storage. This separation is the central design decision — the catalog knows what files exist, what columns they contain, and what value ranges they hold, but it never stores the data itself. In this post, we’ll go from running DuckLake locally to running it with PostgreSQL and MinIO, look at how partitioning organizes data in S3, and see how DuckLake’s query engine prunes files and pushes predicates down to avoid reading data it doesn’t need.