OpenTelemetry in Go — Traces, Metrics, Logs and the Collector

OpenTelemetry gives you traces, metrics, and logs from a single instrumentation library with a vendor-neutral export path. This post walks through adding OpenTelemetry to a Go HTTP server from scratch — setting up providers, auto-instrumenting HTTP, creating manual spans, recording custom metrics, emitting correlated logs, configuring the Collector, and running the full stack with Docker Compose. It finishes with a side-by-side comparison of the OTel metrics API against the Prometheus client library.

What OpenTelemetry Provides

OpenTelemetry (OTel) standardises three types of telemetry signals. Traces track a single request as it flows through your system — each operation within the trace is a span with a name, timing, attributes, and a parent span forming a tree. Metrics are numerical measurements collected over time — counters, histograms, gauges. Logs are timestamped structured records that carry trace context automatically, so each log record is tagged with the trace ID and span ID of the request that produced it.

The key benefit is correlation. Because all three signals share the same trace ID, you don’t need to manually generate and thread a correlation ID through every service and log call — OTel does it automatically. A log line, a metric exemplar, and a span from the same request all carry the same trace ID, so you can jump between them freely.

OTel is a vendor-neutral standard. You instrument once and send the data to whatever backend you want: Jaeger for traces, Prometheus for metrics, Datadog for everything.

Component	What it does
API	Interfaces your code programs against (otel.Tracer(), otel.Meter())
SDK	Implements the API — manages span lifecycle, metric aggregation, log batching
Exporters	Send data to backends (Jaeger, Prometheus, Datadog, etc.)
Collector	A standalone proxy that receives, processes, and exports telemetry
Instrumentation libraries	Auto-instrument common frameworks (HTTP, gRPC, databases)

In this post, we will go through an example straightforward application: your application uses the OTel SDK to create traces, metrics, and logs. The SDK sends this data via OTLP (the OpenTelemetry Protocol) to a Collector, which routes it to the appropriate backends.

┌──────────────┐     OTLP/HTTP       ┌───────────────┐
│   Your App   │ ──────────────────> │  OTel         │
│              │                     │  Collector    │
│  OTel SDK    │                     │               │
│  + Exporters │                     │  receivers    │
└──────────────┘                     │  processors   │
                                     │  exporters    │
                                     └───┬───────┬───┘
                                         │       │
                                    ┌────▼───┐ ┌──▼────────┐
                                    │Jaeger  │ │Prometheus │
                                    │(traces)│ │(metrics)  │
                                    └────────┘ └───────────┘

Full Source Code

https://gitlab.com/kimserey.lam/otel-learn

The Bookstore API

We start with a clean three-layer bookstore API with zero instrumentation. The code uses a simple architecture that makes the OTel additions easy to follow:

cmd/bookstore/main.go        # Entry point
internal/
  model/book.go              # The Book data structure
  store/books.go             # In-memory data store
  service/books.go           # Business logic
  handler/books.go           # HTTP handlers

The Book model is minimal:

type Book struct {
    ID     string `json:"id"`
    Title  string `json:"title"`
    Author string `json:"author"`
    Year   int    `json:"year"`
}

The store layer is an in-memory map[string]Book with a sync.RWMutex for thread safety, exposing List(ctx), Get(ctx, id), and Create(ctx, book). Every method takes a context.Context as its first argument — standard Go practice, but also critical for OTel since the context carries the active trace span.

The handler layer registers four routes using Go 1.22’s enhanced http.ServeMux pattern syntax:

Route	Method	Description
GET /books	listBooks	Returns all books as JSON
GET /books/{id}	getBook	Returns one book, or 404
POST /books	createBook	Creates a book from JSON body
GET /health	health	Health check

The entry point wires everything together:

func main() {
    bookStore := store.New()
    bookService := service.New(bookStore)
    bookHandler := handler.New(bookService)

    mux := http.NewServeMux()
    bookHandler.RegisterRoutes(mux)

    srv := &http.Server{Addr: ":8080", Handler: mux}
    srv.ListenAndServe()
}

The server works, but we have zero visibility into how long each request takes, which internal operations are slow, how many books are being created, or what happened when a request failed.

Adding Traces

Tracing lets you see the full journey of a request through your application. We add both auto-instrumentation (the HTTP layer instruments itself) and manual instrumentation (we create spans in our business logic).

TracerProvider Setup

Before you can create spans, you need a TracerProvider — the SDK component that manages span lifecycle and exports them:

func setupTracerProvider(ctx context.Context, res *resource.Resource) (*sdktrace.TracerProvider, error) {
    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint(stripScheme(endpoint())),
        otlptracehttp.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
    )
    return tp, nil
}

The exporter sends spans over HTTP using OTLP. WithBatcher batches spans before exporting for efficiency. The resource identifies this service — service.name=bookstore, service.version=0.1.0. The provider is registered globally with otel.SetTracerProvider(tp).

Auto-Instrumentation with otelhttp

Wrap your http.Handler to get automatic HTTP tracing:

otelHandler := otelhttp.NewHandler(mux, "bookstore")
srv := &http.Server{Addr: ":8080", Handler: otelHandler}

This single line gives you a root span for every incoming HTTP request, automatic span attributes (http.method, http.route, http.status_code, url.path), 5xx responses marked as Error, and context propagation via the traceparent header so distributed traces work across services.

Manual Instrumentation

To see what happens inside your application, create manual spans. Get a tracer, then call Start:

var tracer = otel.Tracer("bookstore/store")

func (s *BookStore) Get(ctx context.Context, id string) (model.Book, error) {
    ctx, span := tracer.Start(ctx, "BookStore.Get")
    defer span.End()

    span.SetAttributes(attribute.String("book.id", id))

    // ... business logic ...
}

The returned ctx carries the new span. When you pass this ctx to a downstream function, that function’s span becomes a child — building the trace tree. Attributes are key-value pairs that add context to a span and show up in Jaeger, letting you search for traces by attribute value.

When an operation fails, record the error on the span:

if !ok {
    err := fmt.Errorf("book not found: %s", id)
    span.RecordError(err)
    span.SetStatus(codes.Error, "not found")
    return model.Book{}, err
}

RecordError adds an event with the error message and stack trace. SetStatus(codes.Error, ...) marks the span red in Jaeger so you can spot failures at a glance.

Trace Structure

After instrumentation, a GET /books/1 request produces this trace:

HTTP GET /books/1              [=====================]  root span (otelhttp)
  └── BookService.GetBook        [=================]    service span (manual)
       └── BookStore.Get            [============]       store span (manual)
            book.id = "1"

Each span has its own timing, attributes, and status. Spans don’t have to be nested — a parent span can have multiple children that run one after the other:

HTTP POST /books                 [=======================================]
  └── BookStore.Save               [============]
  └── NotifyService.Send                           [============]

In Jaeger’s trace view, traces are displayed as an icicle chart: the root span sits at the top, child spans cascade downward, and time flows left to right.

Context Propagation

The context.Context is the thread that ties spans together. Every function in the chain must accept ctx context.Context as the first parameter and pass the ctx returned by tracer.Start() to downstream calls. If you break the context chain (e.g., using context.Background() instead of the request context), the child span becomes a separate trace.

For cross-service propagation, OTel uses the W3C traceparent HTTP header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  │                                │                │
              │  │                                │                └─ trace flags (01 = sampled)
              │  │                                └─ parent span ID (8 bytes)
              │  └─ trace ID (16 bytes)
              └─ version (always 00)

When Service A calls Service B, otelhttp injects the header with A’s current trace ID and span ID. Service B reads it, creates a new span linked as a child, and both services end up in the same trace. The propagator is set up with:

prop := propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
)
otel.SetTextMapPropagator(prop)

Adding Metrics

Traces show you individual requests. Metrics show you aggregate behaviour over time — request rates, error rates, latency percentiles. They’re lightweight, cheap to collect, and the foundation for dashboards and alerting.

MeterProvider Setup

func setupMeterProvider(ctx context.Context, res *resource.Resource) (*sdkmetric.MeterProvider, error) {
    exporter, err := otlpmetrichttp.New(ctx,
        otlpmetrichttp.WithEndpoint(stripScheme(endpoint())),
        otlpmetrichttp.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    mp := sdkmetric.NewMeterProvider(
        sdkmetric.WithReader(sdkmetric.NewPeriodicReader(
            exporter,
            sdkmetric.WithInterval(10*time.Second),
        )),
        sdkmetric.WithResource(res),
    )
    return mp, nil
}

The PeriodicReader collects and exports metrics every 10 seconds. The OTLP HTTP exporter sends metrics to the Collector, same as traces.

Instrument Types

OTel defines several instrument types:

Instrument	Use case	Example
Counter	Values that only increase	Total books created
UpDownCounter	Values that increase and decrease	Current books in store
Histogram	Distribution of values	Request duration
Gauge	Point-in-time snapshot of a value	CPU usage, queue depth

Custom Metrics

In the service layer, we create three custom instruments:

var meter = otel.Meter("bookstore/service")

booksCreated, _ := meter.Int64Counter("books.created",
    metric.WithDescription("Total number of books created"),
    metric.WithUnit("{book}"),
)

booksCount, _ := meter.Int64UpDownCounter("books.count",
    metric.WithDescription("Current number of books in the store"),
    metric.WithUnit("{book}"),
)

operationDuration, _ := meter.Float64Histogram("books.operation.duration",
    metric.WithDescription("Duration of service-layer operations"),
    metric.WithUnit("s"),
)

Recording is straightforward — increment the counter when a book is created:

func (svc *BookService) CreateBook(ctx context.Context, book model.Book) model.Book {
    // ... create the book ...
    svc.booksCreated.Add(ctx, 1)
    svc.booksCount.Add(ctx, 1)
    return created
}

Measure operation duration with the histogram:

func (svc *BookService) GetBook(ctx context.Context, id string) (model.Book, error) {
    start := time.Now()
    defer func() {
        svc.operationDuration.Record(ctx, time.Since(start).Seconds(),
            metric.WithAttributes(attribute.String("operation", "GetBook")),
        )
    }()
    // ... business logic ...
}

The operation attribute becomes a label in Prometheus, letting you break down duration by operation type.

Auto-Instrumented HTTP Metrics

The otelhttp.NewHandler middleware also records HTTP metrics automatically:

Metric	Type	Description
http.server.request.duration	Histogram	Request latency
http.server.request.body.size	Histogram	Request body size
http.server.response.body.size	Histogram	Response body size
http.server.active_requests	UpDownCounter	Currently processing

These come for free — no code changes needed.

Push vs Pull

In our architecture, the app pushes to the Collector via OTLP, and then the Collector exposes a Prometheus-compatible scrape endpoint. Prometheus scrapes the Collector, not the app directly. This decouples the app from Prometheus.

OTel uses dots in metric names (e.g., books.created), but Prometheus uses underscores (e.g., bookstore_books_created_total). The Collector’s Prometheus exporter handles this conversion automatically.

Adding Logs

Logs are the oldest form of telemetry, but OTel gives them a superpower: trace correlation. Every log record is automatically tagged with the trace ID and span ID of the active request, letting you jump between logs and traces seamlessly.

LoggerProvider Setup

func setupLoggerProvider(ctx context.Context, res *resource.Resource) (*sdklog.LoggerProvider, error) {
    exporter, err := otlploghttp.New(ctx,
        otlploghttp.WithEndpoint(stripScheme(endpoint())),
        otlploghttp.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    lp := sdklog.NewLoggerProvider(
        sdklog.WithProcessor(sdklog.NewBatchProcessor(exporter)),
        sdklog.WithResource(res),
    )
    return lp, nil
}

The slog Bridge

Go 1.21 introduced log/slog — a structured logging package in the standard library. Rather than using OTel’s log API directly, we use the otelslog bridge, which makes slog emit log records through the OTel pipeline:

logger = otelslog.NewLogger("bookstore")
slog.SetDefault(logger)

Now every slog.Info(), slog.Warn(), slog.Error() call sends log records to the OTel Collector via OTLP.

Writing Logs with Trace Correlation

The key is slog.InfoContext(ctx, ...) — the Context variant. The bridge extracts the active span from the context and attaches its trace ID and span ID to the log record:

func (svc *BookService) CreateBook(ctx context.Context, book model.Book) model.Book {
    slog.InfoContext(ctx, "creating book",
        "book.title", book.Title,
        "book.author", book.Author,
    )

    created := svc.store.Create(ctx, book)

    slog.InfoContext(ctx, "book created",
        "book.id", created.ID,
        "book.title", created.Title,
    )
    return created
}

In the Collector’s debug output, each log record includes TraceID and SpanID, linking the log record to the exact trace and span that produced it. If you use slog.Info() without context, the log record won’t have trace/span IDs — always use the context variants (InfoContext, WarnContext, ErrorContext) when you’re inside a request handler.

The Three-Way Correlation

With all three signals in place, they’re connected:

• Trace -> Logs: Each log record carries the trace ID, so you can find all logs for a specific trace
• Logs -> Trace: From a log line, you can jump to the full trace to see the request flow
• Metrics + Traces: They share the same resource attributes (service.name, service.version), linking them to the same service

The OTel Collector

The OTel Collector is a vendor-neutral proxy that sits between your application and your observability backends. It receives telemetry, processes it, and exports it to one or more destinations.

Why Use a Collector

You could export directly from your app to Jaeger and Prometheus. But the Collector gives you:

• Decoupling: Switch from Jaeger to Tempo? Change the Collector config, not your code.
• Processing: Batch data, drop noisy spans, add attributes, sample traces — all without touching your app.
• Fan-out: Send the same data to multiple backends.
• Buffering: The Collector can buffer data during backend outages.

The Pipeline Model

The Collector is built around pipelines. Each pipeline has three stages:

Receivers ──> Processors ──> Exporters

You define separate pipelines for each signal type (traces, metrics, logs).

Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: bookstore

  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

The otlp receiver accepts data over both gRPC (port 4317) and HTTP (port 4318). Our Go app sends data over HTTP to port 4318.

The batch processor groups data points together before sending them to exporters — it sends a batch when it reaches 1024 items or every 5 seconds, whichever comes first.

For exporters: modern Jaeger natively accepts OTLP on port 4317. The Prometheus exporter doesn’t push to Prometheus — it exposes a /metrics endpoint on port 8889 that Prometheus scrapes. The namespace: bookstore prefix is added to all metric names. The debug exporter prints telemetry to stdout.

Each signal gets its own pipeline. Notice that traces go to two exporters (otlp/jaeger and debug) — the Collector fans out automatically.

There are two Collector distributions: core (minimal) and contrib (includes community-contributed components). We use contrib because the Prometheus exporter is not in the core distribution.

Running Everything with Docker Compose

Docker Compose runs four services that form a complete observability stack.

services:
  bookstore:
    build:
      context: .
      dockerfile: docker/Dockerfile
    ports:
      - "8080:8080"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.117.0
    volumes:
      - ./config/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "8889:8889"   # Prometheus metrics endpoint
    depends_on:
      - jaeger

  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686" # Jaeger UI

  prometheus:
    image: prom/prometheus:v3.1.0
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"   # Prometheus UI
    depends_on:
      - otel-collector

The bookstore app uses the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to reach the Collector via Docker network hostname otel-collector. The Collector uses the contrib image and mounts the config file. Jaeger’s all-in-one image runs the collector, query, and UI in a single container with native OTLP enabled. Prometheus scrapes the Collector’s Prometheus exporter every 15 seconds.

Network connectivity uses Docker Compose’s default network where services reach each other by name:

bookstore → otel-collector:4318        (OTLP HTTP)
otel-collector → jaeger:4317           (OTLP gRPC for traces)
prometheus → otel-collector:8889       (HTTP scrape for metrics)

The Dockerfile uses a multi-stage build — a Go 1.25 builder stage compiles a static binary with CGO_ENABLED=0, and a minimal Alpine runtime stage copies just the binary (~8MB):

FROM golang:1.25-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /bookstore ./cmd/bookstore

FROM alpine:3.20
COPY --from=builder /bookstore /bookstore
EXPOSE 8080
CMD ["/bookstore"]

Build and start everything with docker compose up --build. Once all services are running, generate some traffic:

curl http://localhost:8080/health

curl -X POST http://localhost:8080/books \
  -H "Content-Type: application/json" \
  -d '{"title": "Observability Engineering", "author": "Majors, Fong-Jones & Miranda", "year": 2022}'

curl http://localhost:8080/books
curl http://localhost:8080/books/1
curl http://localhost:8080/books/999  # 404 — generates an error span

Viewing Your Telemetry

Traces in Jaeger

Open http://localhost:16686. Select bookstore in the Service dropdown and click Find Traces. Click on a trace for GET /books/{id} to see the waterfall view:

bookstore: GET /books/{id}              ████████████████████  5.2ms
  └── bookstore: BookService.GetBook      ██████████████████  4.8ms
       └── bookstore: BookStore.Get          █████████████████  4.5ms

Each bar is a span. Click on a span to see its tags/attributes (book.id, http.method, http.route, http.status_code), logs/events (error events if the span recorded an error), process metadata (service.name=bookstore), and timing. Error traces have red indicators — the GET /books/999 request shows a span with status=Error.

In the search panel, add a tag filter like book.id = 1 to find all traces related to that book.

Metrics in Prometheus

Open http://localhost:9090. Type queries in the Expression box:

bookstore_books_created_total

bookstore_books_count

histogram_quantile(0.95, rate(bookstore_books_operation_duration_bucket[5m]))

Auto-instrumented HTTP metrics come for free:

rate(bookstore_http_server_request_duration_seconds_count[5m])

histogram_quantile(0.95, rate(bookstore_http_server_request_duration_seconds_bucket[5m]))

sum by (http_route, http_request_method) (
  rate(bookstore_http_server_request_duration_seconds_count[5m])
)

Error rate (4xx + 5xx responses):

sum(rate(bookstore_http_server_request_duration_seconds_count{http_response_status_code=~"[45].."}[5m]))
/
sum(rate(bookstore_http_server_request_duration_seconds_count[5m]))

Logs in Collector Output

Since we export logs to the debug exporter, they appear in the Collector’s stdout with docker compose logs -f otel-collector:

LogRecord #0
ObservedTimestamp: 2024-01-15 10:30:42.123456789 +0000 UTC
SeverityText: INFO
Body: Str(book created)
Attributes:
  -> book.id: Str(1)
  -> book.title: Str(The Go Programming Language)
TraceID: abc123def456...
SpanID: 789xyz...

The TraceID and SpanID link the log record to the exact trace and span that produced it. In production with Grafana and Loki, you could click a log line and jump directly to its trace in Tempo or Jaeger.

OpenTelemetry vs Prometheus Client

You might wonder why we used OpenTelemetry for metrics instead of the Prometheus client library directly. After all, our metrics end up in Prometheus anyway.

Feature Comparison

	Prometheus client	OpenTelemetry
Scope	Metrics only	Metrics + traces + logs
Export model	Pull — exposes /metrics endpoint	Push — sends OTLP to a Collector
Backend lock-in	Tied to Prometheus	Vendor-neutral
Trace correlation	None	Built-in

Defining a Counter

With the Prometheus client:

var booksCreated = prometheus.NewCounter(prometheus.CounterOpts{
    Namespace: "bookstore",
    Name:      "books_created_total",
    Help:      "Total number of books created",
})

func init() {
    prometheus.MustRegister(booksCreated)
}

With OpenTelemetry:

var meter = otel.Meter("bookstore/service")

booksCreated, _ := meter.Int64Counter("books.created",
    metric.WithDescription("Total number of books created"),
    metric.WithUnit("{book}"),
)

Similar amount of code. The Prometheus version requires a global init() to register the metric. OTel metrics are registered implicitly when you create them from a Meter.

Recording a Histogram

With the Prometheus client:

opDuration.WithLabelValues("GetBook").Observe(time.Since(start).Seconds())

With OpenTelemetry:

svc.operationDuration.Record(ctx, time.Since(start).Seconds(),
    metric.WithAttributes(attribute.String("operation", "GetBook")),
)

Prometheus uses WithLabelValues() with label names declared upfront. OTel uses attribute.String() at record time — more flexible, but you need to be careful about cardinality. OTel always takes a ctx parameter, which is how metrics get correlated with traces.

Exposing Metrics

This is where the architectures diverge most. The Prometheus client exposes a /metrics endpoint that Prometheus scrapes directly:

mux.Handle("/metrics", promhttp.Handler())

OTel pushes metrics to the Collector, and the Collector exposes the Prometheus endpoint:

┌─────────────┐  OTLP push  ┌────────────────┐  scrape  ┌────────────┐
│  Your App   │ ──────────> │ OTel Collector │ <─────── │ Prometheus │
└─────────────┘             └────────────────┘          └────────────┘

The extra Collector hop adds a component to run, but decouples your app from the backend entirely. If you later switch from Prometheus to Grafana Cloud or Datadog, you change the Collector config — zero code changes.

When to Use Which

Use the Prometheus client when you only need metrics, you’re running a simple service with the fewest dependencies, and your infrastructure is 100% Prometheus. Use OpenTelemetry when you want metrics and traces, vendor flexibility, trace correlation in a multi-service architecture, or your organisation is standardising on OTel.

You can also use OTel with Prometheus’s pull model directly — the OTel Prometheus exporter serves a /metrics endpoint from the OTel SDK without needing a Collector for metrics:

promExporter, _ := prometheus.New()
mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(promExporter))

Naming Translation

When OTel metrics land in Prometheus, names are converted automatically:

OTel metric name	Prometheus name
books.created (Counter)	bookstore_books_created_total
books.count (UpDownCounter)	bookstore_books_count
books.operation.duration (Histogram)	bookstore_books_operation_duration_seconds

Dots become underscores. Counters get _total. Duration histograms get _seconds. The bookstore prefix comes from the namespace in the Collector config.