Deploying DuckLake on AWS with CloudFormation

DuckLake needs two pieces of infrastructure: an S3 bucket for Parquet data files and a PostgreSQL database for the metadata catalog. This post walks through standing up that infrastructure on AWS using CloudFormation — a single YAML template that creates all the resources as a unit. Along the way, we cover KMS encryption, bucket policies, Aurora PostgreSQL configuration, IAM roles for workload access, and the deployment workflow.

What CloudFormation Gives You

CloudFormation (CF) is AWS’s infrastructure-as-code tool. You write a YAML template declaring the resources you want, and CF creates, updates, or deletes them as a unit called a stack. The key concepts:

Concept	What it is
Template	A YAML file declaring resources
Stack	A named collection of AWS resources created from a template
Logical ID	The label for a resource in your template (e.g. KmsKey, DucklakeStorage)
Physical ID	The actual AWS resource name or ARN — either set explicitly or auto-generated
Parameters	Inputs to the template (like EnvironmentName, DatabasePassword)
Outputs	Values exported from the stack for other stacks or humans to reference

Two things set CF apart from tools like Terraform: no state file — CF stores state in AWS itself, so there’s no S3 backend, no locking, no state corruption — and automatic rollback on failure, so you never end up with a half-deployed stack.

The DuckLake Stack

DuckLake’s architecture splits metadata from data. The catalog (PostgreSQL) tracks table schemas, snapshots, column statistics, and file pointers. The data files (Parquet) sit in S3. Our CF template needs to create both sides plus the security layer that ties them together.

Here’s what the stack looks like end to end:

Resource	Type	Purpose
KmsKey	KMS Key	Encrypts S3 objects and RDS data at rest
DucklakeStorage	S3 Bucket	Parquet file storage
DucklakeStoragePolicy	Bucket Policy	Deny rules: no HTTP, no unencrypted uploads
ParameterGroup	RDS Parameter Group	SSL and replication settings
DucklakeCatalog	RDS Cluster	Aurora PostgreSQL for the metadata catalog
DucklakeCatalogInstance	RDS Instance	Compute instance for the cluster
ServicePolicy	IAM Policy	Grants S3 read/write and KMS encrypt/decrypt
ServiceRole	IAM Role	Assumed by workloads to get AWS access

Let’s walk through each piece.

Encryption: KMS Key

Both S3 and Aurora share a single KMS key for encryption at rest:

KmsKey:
  Type: AWS::KMS::Key
  Properties:
    Description: !Sub 'DuckLake encryption key - ${EnvironmentName}'
    EnableKeyRotation: true
    KeyPolicy:
      Version: '2012-10-17'
      Statement:
        - Sid: EnableRootAccess
          Effect: Allow
          Principal:
            AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
          Action: 'kms:*'
          Resource: '*'

The KeyPolicy grants the AWS account root full access to the key. This is required — without it, no IAM policy can grant access to the key. EnableKeyRotation is a security best practice: AWS automatically rotates the backing key material annually.

!Sub is a CF intrinsic function that substitutes variables into a string. ${EnvironmentName} comes from a template parameter, and ${AWS::AccountId} is a pseudo-parameter that CF resolves to the current account ID.

Object Storage: S3 Bucket

The bucket holds all Parquet files that DuckLake writes:

DucklakeStorage:
  Type: AWS::S3::Bucket
  Properties:
    BucketName: !Sub ducklake-storage-${EnvironmentName}
    VersioningConfiguration:
      Status: Enabled
    BucketEncryption:
      ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: aws:kms
            KMSMasterKeyID: !Ref KmsKey

!Ref KmsKey references the KMS key by its logical ID — CF resolves this to the key’s ID at deploy time. Versioning is enabled so you can recover from accidental deletes or overwrites.

Bucket Policy: Security Guardrails

DucklakeStoragePolicy:
  Type: AWS::S3::BucketPolicy
  Properties:
    Bucket: !Ref DucklakeStorage
    PolicyDocument:
      Statement:
        - Sid: DenyInsecureTransport
          Effect: Deny
          Principal: '*'
          Action: 's3:*'
          Resource:
            - !GetAtt DucklakeStorage.Arn
            - !Sub '${DucklakeStorage.Arn}/*'
          Condition:
            Bool:
              aws:SecureTransport: 'false'
        - Sid: DenyUnEncryptedObjectUploads
          Effect: Deny
          Principal: '*'
          Action: 's3:PutObject'
          Resource: !Sub '${DucklakeStorage.Arn}/*'
          Condition:
            StringNotEquals:
              s3:x-amz-server-side-encryption: 'aws:kms'

These are deny rules on the bucket itself — enforced regardless of who’s calling. Even if an IAM policy allows s3:PutObject, the bucket policy still blocks unencrypted uploads or HTTP requests. This is a different layer from the IAM service policy, which defines what a specific role is allowed to do. The bucket policy sets guardrails that no caller can bypass.

DeletionPolicy

For production, add DeletionPolicy: Retain to the bucket so it survives stack deletion — prevents accidental data loss. For dev environments, DeletionPolicy: Delete cleans up everything when the stack is torn down.

Metadata Catalog: Aurora PostgreSQL

Aurora PostgreSQL serves as DuckLake’s metadata catalog. This requires three resources:

Parameter Group

DucklakeCatalogParameterGroup:
  Type: AWS::RDS::DBClusterParameterGroup
  Properties:
    Family: aurora-postgresql17
    Parameters:
      ssl: '1'
      rds.force_ssl: '1'
      rds.logical_replication: '1'

Cluster-level settings: force SSL on all connections and enable logical replication (useful if you ever want to stream changes out of the catalog).

Cluster

DucklakeCatalog:
  Type: AWS::RDS::DBCluster
  Properties:
    DBClusterIdentifier: !Sub '${EnvironmentName}-ducklake-catalog'
    Engine: aurora-postgresql
    EngineVersion: '17.4'
    DatabaseName: ducklake_catalog
    MasterUsername: postgres
    MasterUserPassword: !Ref DucklakeCatalogPassword
    StorageEncrypted: true
    KmsKeyId: !GetAtt KmsKey.Arn
    DBSubnetGroupName: !Ref DBSubnetGroup
    VpcSecurityGroupIds:
      - !Ref RDSSecurityGroup

The cluster definition ties everything together — engine version, encryption with the shared KMS key, networking via subnet group and security group. !GetAtt KmsKey.Arn pulls the ARN attribute from the KMS key, as opposed to !Ref KmsKey which returns the key ID. Different CF functions return different things for the same resource.

The networking references (DBSubnetGroupName, VpcSecurityGroupIds) point to resources that define which subnets and what traffic rules apply. Aurora requires subnets in at least two availability zones for high availability.

Instance

DucklakeCatalogInstance:
  Type: AWS::RDS::DBInstance
  Properties:
    DBClusterIdentifier: !Ref DucklakeCatalog
    DBInstanceClass: db.t4g.medium
    Engine: aurora-postgresql

The instance is the actual compute. Aurora separates storage from compute — the cluster owns the storage layer and the instance provides the processing power. For a metadata catalog (which handles schema lookups and statistics, not heavy query workloads), db.t4g.medium is a reasonable starting point.

Access Control: IAM Policy and Role

Service Policy — What the Workload Can Do

ServicePolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    PolicyDocument:
      Statement:
        - Effect: Allow
          Action:
            - s3:GetObject
            - s3:PutObject
            - s3:DeleteObject
            - s3:ListBucket
          Resource:
            - !GetAtt DucklakeStorage.Arn
            - !Sub '${DucklakeStorage.Arn}/*'
        - Effect: Allow
          Action:
            - kms:Decrypt
            - kms:Encrypt
            - kms:GenerateDataKey
          Resource: !GetAtt KmsKey.Arn

Two resource entries for S3: the bucket ARN itself (for bucket-level operations like ListBucket) and the bucket ARN with /* (for object-level operations like GetObject and PutObject). This is a common gotcha — missing either one causes permission errors that are hard to debug. KMS permissions are needed because the bucket uses KMS encryption; every read decrypts and every write encrypts.

Service Role — Who Can Assume It

ServiceRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Statement:
        - Action: sts:AssumeRoleWithWebIdentity
          Effect: Allow
          Principal:
            Federated: !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/${OIDCProvider}'
    ManagedPolicyArns:
      - !Ref ServicePolicy

The role’s trust policy determines who can assume it. In this example, it uses OIDC federation — workloads running in Kubernetes (EKS) present an OIDC token, AWS STS validates it against the trusted provider, and returns temporary credentials. The flow at runtime:

1. Workload starts
   └─ Platform injects an OIDC token

2. Code calls s3.PutObject(...)
   └─ AWS SDK detects the OIDC token

3. SDK calls AWS STS: AssumeRoleWithWebIdentity
   └─ STS validates the token against the trusted OIDC provider
   └─ Returns temporary credentials (expire in 1hr)

4. SDK uses temporary credentials to call S3
   └─ S3 checks the ServicePolicy

5. Credentials expire → SDK repeats steps 3-4 automatically

No static access keys stored anywhere. The trust chain is: workload platform trusts the service account, which can assume the IAM role, which has S3 and KMS permissions.

For non-Kubernetes deployments, the trust policy would change — an EC2 instance profile, an ECS task role, or a Lambda execution role — but the service policy stays the same.

CF Intrinsic Functions

A quick reference for the functions used throughout this template:

Function	What it does	Example
!Ref	Returns the ID of a resource or parameter value	!Ref KmsKey → the key ID
!Sub	String interpolation	!Sub 'bucket-${EnvironmentName}' → bucket-dev
!GetAtt	Get a specific attribute of a resource	!GetAtt KmsKey.Arn → the key’s ARN
Fn::ImportValue	Import an output from another stack	Pull in networking resources from a shared stack

!Ref and !GetAtt look similar but return different things. !Ref KmsKey returns the key ID. !GetAtt KmsKey.Arn returns the full ARN. Which one you need depends on the property you’re setting — KMSMasterKeyID takes an ID, KmsKeyId takes an ARN.

Outputs and Cross-Stack References

Outputs:
  BucketArn:
    Value: !GetAtt DucklakeStorage.Arn
    Export:
      Name: !Sub '${AWS::StackName}-bucket-arn'
  CatalogEndpoint:
    Value: !GetAtt DucklakeCatalog.Endpoint.Address
    Export:
      Name: !Sub '${AWS::StackName}-catalog-endpoint'

Outputs serve two purposes: they’re human-readable (visible via aws cloudformation describe-stacks) and they enable cross-stack references (other stacks import them via Fn::ImportValue). Never put passwords in outputs — they’re visible to anyone who can call describe-stacks.

Stack Naming

The stack name becomes the prefix for auto-generated resource names. If you deploy multiple environments to the same AWS account, include the environment in the stack name:

# Bad: no environment context
aws cloudformation deploy --stack-name ducklake-storage ...
# → ducklake-storage-ducklakecatalog-ib2dzx4p4vs3

# Good: environment is clear
aws cloudformation deploy --stack-name ducklake-storage-dev ...
# → ducklake-storage-dev-ducklakecatalog-xxxx

Deploying

Validate First

# Install the CF linter
brew install cfn-lint

# Lint — catches syntax errors, bad refs, best practice violations
cfn-lint cloudformation/ducklake-storage.yml

# Basic AWS syntax check
aws cloudformation validate-template \
  --template-body file://cloudformation/ducklake-storage.yml

Deploy the Stack

aws cloudformation deploy \
  --template-file cloudformation/ducklake-storage.yml \
  --stack-name ducklake-storage-dev \
  --parameter-overrides \
    EnvironmentName=dev \
    DucklakeCatalogPassword=<password> \
  --capabilities CAPABILITY_NAMED_IAM

CAPABILITY_NAMED_IAM is required because the template creates IAM resources. CF makes you explicitly acknowledge this.

Inspect After Deployment

# List all resources in the stack
aws cloudformation describe-stack-resources \
  --stack-name ducklake-storage-dev --output table

# Check outputs
aws cloudformation describe-stacks \
  --stack-name ducklake-storage-dev \
  --query 'Stacks[0].Outputs'

Tear Down

# Empty the bucket first — S3 won't delete a non-empty bucket
aws s3 rm s3://ducklake-storage-dev --recursive

# Delete the stack
aws cloudformation delete-stack --stack-name ducklake-storage-dev

# Wait for completion
aws cloudformation wait stack-delete-complete --stack-name ducklake-storage-dev

Summary

One CF template, one deploy command, and you get everything DuckLake needs on AWS: an encrypted S3 bucket for Parquet files, an Aurora PostgreSQL cluster for the metadata catalog, a KMS key shared by both, bucket policies that enforce HTTPS and encryption, and an IAM role that gives workloads scoped access to S3 and KMS without static credentials. CF manages the dependency graph between resources, rolls back on failure, and stores all state in AWS — no external state files to manage.