AWS Credential Chain in DuckDB

When DuckDB reads or writes Parquet files in S3, it needs AWS credentials. Rather than hardcoding access keys, you can tell DuckDB to discover credentials at runtime using the same chain the AWS SDK uses — environment variables, IRSA tokens, SSO profiles, ECS task roles, or EC2 instance metadata. This post covers how the credential chain works, what each step looks for, and how it connects to DuckLake in both local development and production.

Static Keys vs. Credential Chain

DuckDB gives you two ways to configure S3 access. The first is explicit — you provide the keys directly:

CREATE SECRET (
  TYPE s3,
  KEY_ID 'AKIA...',
  SECRET 'wJal...',
  REGION 'us-east-1'
);

No discovery happens. DuckDB uses exactly what you gave it. This is what you’d use with a local S3-compatible store like MinIO or SeaweedFS during development.

The second is the credential chain — DuckDB searches multiple sources in order and uses the first one that works:

CREATE SECRET (
  TYPE s3,
  PROVIDER credential_chain,
  REGION 'us-east-1'
);

This is what you’d use in production where credentials come from the environment, not from your code.

The Chain, Step by Step

The credential chain checks sources in a fixed order and stops at the first one that provides valid credentials:

Step	Source	What it looks for
1	Environment variables	AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY (+ optional AWS_SESSION_TOKEN)
2	Web Identity Token (IRSA)	AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE → calls STS AssumeRoleWithWebIdentity
3	SSO / Profile config	~/.aws/config + ~/.aws/credentials
4	ECS task role	AWS_CONTAINER_CREDENTIALS_RELATIVE_URI → calls the ECS metadata endpoint
5	EC2 instance metadata (IMDS)	http://169.254.169.254/latest/meta-data/iam/security-credentials/

If none succeed, the secret creation fails.

In practice, which step fires depends on where your code runs:

• Your laptop → step 3 (your ~/.aws config from aws sso login or aws configure)
• A Kubernetes pod with IRSA → step 2 (the platform injects the env vars)
• An ECS/Fargate task → step 4 (task role metadata)
• A bare EC2 instance → step 5 (instance profile)

You don’t pick the step — you just use PROVIDER credential_chain and the runtime environment determines which source matches.

IRSA: How Kubernetes Pods Get AWS Credentials

IRSA (IAM Roles for Service Accounts) is the standard way to give Kubernetes pods AWS permissions without static keys. When a ServiceAccount is annotated with an IAM role:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-service-role

AWS mutates the pod spec at admission time to inject two environment variables:

AWS_ROLE_ARN=arn:aws:iam::123456789012:role/my-service-role
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

The token file contains a JWT that kubelet keeps rotated (default 24-hour expiry, refreshed at ~80% TTL). When the credential chain hits step 2, it reads this token, calls STS AssumeRoleWithWebIdentity, and gets back temporary credentials — an AccessKeyId, SecretAccessKey, and SessionToken — that typically last one hour.

Two Layers of Rotation

Both layers are automatic — no application code needed:

1. Kubelet rotates the JWT token file on disk (~every 24 hours)
2. The AWS SDK / DuckDB refreshes the STS session when it expires (~every 1 hour)

Your application just calls S3 operations. The SDK handles the token exchange and credential refresh transparently.

Verifying IRSA in a Running Pod

# See which AWS env vars are set — tells you which chain step will fire
env | grep -E '^AWS_'

# Expected IRSA output:
# AWS_ROLE_ARN=arn:aws:iam::123456789012:role/my-service-role
# AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
# AWS_DEFAULT_REGION=us-east-1

# Decode the JWT to check expiry
cat $AWS_WEB_IDENTITY_TOKEN_FILE | cut -d. -f2 | base64 -d 2>/dev/null
# Look for the "exp" field (Unix timestamp)

How DuckDB Implements the Chain

The credential chain logic lives in DuckDB’s aws extension (the duckdb/duckdb-aws repository), not in core DuckDB. Core DuckDB maps PROVIDER credential_chain to the aws extension via a lookup table:

{"s3/credential_chain", "aws"},
{"gcs/credential_chain", "aws"},
{"r2/credential_chain", "aws"},

When you create a secret with PROVIDER credential_chain, DuckDB auto-loads the aws extension, which handles the STS calls and credential refresh. This means the credential chain also works for GCS and Cloudflare R2 — any S3-compatible storage that DuckDB supports.

Credential Chain with DuckLake

When building a service that uses DuckDB with DuckLake and S3 storage, you typically support two modes based on the environment:

Local development — static keys pointing at a local S3-compatible store:

CREATE SECRET (
  TYPE s3,
  KEY_ID 'minioadmin',
  SECRET 'minioadmin',
  ENDPOINT 'localhost:9000',
  REGION 'us-east-1',
  URL_STYLE 'path',
  USE_SSL false
);

Production — credential chain with IRSA (or whatever the platform provides):

CREATE SECRET (
  TYPE s3,
  PROVIDER credential_chain,
  REGION 'us-east-1'
);

The decision logic is straightforward: if an S3 endpoint is configured (pointing to a local store), use static keys. If no endpoint and no static keys are provided, use the credential chain and let the runtime environment supply credentials.

Credential Chain vs. Other “Chains”

The word “chain” appears in several unrelated contexts. They solve different problems:

Concept	What it does	Where it matters
AWS credential chain	Finds AWS access keys at runtime by checking sources in order	S3 access, STS calls
Certificate chain (TLS)	Verifies server identity via a trust hierarchy (leaf → intermediate → root CA)	TLS connections to PostgreSQL, HTTPS
Keychain (macOS/Linux)	Local credential store on your machine	Local development only

The credential chain is a runtime discovery mechanism. The certificate chain is a trust verification mechanism. They happen to share a name but have nothing else in common.