Observability

Everything here uses the standard library and OpenTelemetry. There is no vendor SDK to swap out: point the OTLP exporter at whatever backend you run.

Logs

Both services log JSON lines with log/slog. Each line inside a request carries request_id, and trace_id and span_id when a trace is active, so you can go from a log line to its trace and back.

• LOG_LEVEL sets the level (debug, info, warn, error; default info).
• LOG_FORMAT is json (default) or text, which is easier to read in a terminal.

Every HTTP request gets one line, request, with method, pattern, status, bytes, duration and the request ID, plus err when the handler failed. A 5xx logs at error level, and a client that left before the response at debug. The route pattern, such as /v1/clients/{id}, is logged instead of the raw path, so IDs in paths don’t explode your log index. The gateway’s access log keeps the path without its query string, so the sign-in codes in mailed links stay out of it.

Request IDs. An incoming X-Request-Id is kept if it is 1 to 64 characters of letters, digits, ., _ or -. Otherwise the service makes a new one. Either way it is echoed in the response and in every problem body.

Redaction. The handler replaces the values of keys that hold credentials with [redacted]: password, secret, token, access_token, refresh_token, authorization, cookie, set-cookie, client_secret, code and code_verifier. Secrets in config use a Secret type that prints [redacted] too.

Traces and metrics

When OTEL_EXPORTER_OTLP_ENDPOINT is set, or one of its per-signal variants, the services export traces and metrics over OTLP/HTTP. When it isn’t, the providers are no-ops and cost nothing. These are the OpenTelemetry SDK’s standard variables, so any collector works.

Incoming HTTP requests get a span named after the method and route, such as GET /v1/clients/{id}. Trace context propagates with the W3C traceparent header and baggage.

Packages record their own metrics with the global meter. The outbox relay, for example, counts delivered and buried messages:

internal/platform/outbox/relay.go

func (r *Relay) init() {
  r.initOnce.Do(func() {
    r.Config = r.Config.withDefaults()
    if r.Logger == nil {
      r.Logger = slog.Default()
    }
    if r.Now == nil {
      r.Now = time.Now
    }
    meter := otel.Meter("github.com/vardius/go-api-boilerplate/internal/platform/outbox")
    // The no-op meter never fails, and a failing real one returns a
    // usable no-op counter, so these errors carry nothing to act on.
    r.delivered, _ = meter.Int64Counter("outbox.delivered", metric.WithDescription("Messages delivered."))
    r.buried, _ = meter.Int64Counter("outbox.buried", metric.WithDescription("Messages given up on after repeated failures."))
  })
}

Locally, the compose.observability.yaml override adds grafana/otel-lgtm, which bundles a collector, Grafana and storage for logs, traces and metrics, and points both services at it. Grafana is then on http://localhost:3000:

make up OBSERVABILITY=1

Health probes

Route	Answers
/livez	200 while the process runs.
/readyz	200 when every readiness check passes and the service isn’t shutting down. Otherwise 503, with a problem body naming the checks that failed or saying the service is draining. A check’s error is logged, not sent.

The readiness checks cover what the service needs to serve, such as its database. Kubernetes probes these routes; compose runs <service> healthcheck, which asks the local /readyz, because the distroless image has no shell or curl.

Shutdown

On SIGINT or SIGTERM a service:

1. flips /readyz to 503, so load balancers stop sending it traffic;
2. waits SHUTDOWN_DELAY (default 5s) for them to notice;
3. shuts down every server and worker concurrently, allowing SHUTDOWN_TIMEOUT (default 20s) for requests in flight to finish;
4. exits non-zero if anything failed to stop cleanly.