Observability

Structured logging, the metrics contract and its backends, trace-ID propagation, and log redaction in nurago

Table of contents

These concerns stay in separate packages so each can be adopted alone: what the service writes (logging), what it counts (metrics), what correlates a request’s records (trace IDs), and what must never reach the output (redaction).

Logging

Application code depends on the standard log/slog and nothing else. The handler behind it comes from either logutil or logsrv.

logutil is the configuration-driven path, on the standard library alone. Config and its options build handlers and loggers from typed settings; ParseLevel, ParseFormat, ValidLevel, and ValidFormat turn runtime configuration strings into those settings and reject the invalid ones. Output is JSON, text console, or discard. The severity model extends the syslog levels (emergency through debug) with a trace level, so a service can log at severities the standard slog levels do not name.

NewSlogHookHandler intercepts every emitted record through a HookFunc. That is how bootstrap increments a metrics counter per log level without the calling code knowing.

logsrv is the same configuration model with a zerolog backend: a native slog.Handler that writes each record’s attributes directly onto a zerolog Event. Use it when you want zerolog’s encoding performance while application code still depends only on log/slog.

The two backends are interchangeable, though their output is not byte-identical. /packages/logsrv/ documents every divergence: the message field name, the position of the injected trace ID, and the value shapes that render differently. Check it before swapping backends under a log-search rule that keys on one of those shapes.

Both handlers filter records before encoding, which repairs two shapes the standard library encodes incorrectly: a group whose members all render nothing (which slog leaves unclosed, producing invalid JSON) and a time.Time outside year 0 to 9999 (which slog writes twice under one key).

Metrics

metrics defines the contract, and reaches no external module. Client abstracts the instrumentation points a service actually uses:

  • SQL opening and database instrumentation
  • inbound HTTP handler instrumentation
  • outbound HTTP round-tripper instrumentation
  • the metrics endpoint handler
  • application counters for log levels and an error taxonomy

Default is a no-op implementation that still returns a working SQL connection from SqlOpen and answers “OK” from its metrics handler, so a service runs without a configured backend and tests run without a live one.

Three backends implement the contract:

  • metrics/prometheus registers Go runtime and process collectors, HTTP server request count, in-flight gauge, and duration, request-size and response-size histograms, HTTP client request count, in-flight gauge and duration, SQL pool stats, and the error counters. Use it when your platform scrapes.
  • metrics/opentel provides metrics and tracing through OpenTelemetry, with otelhttp instrumentation for inbound and outbound HTTP and otelsql for the database. New registers the global tracer, meter, and propagator, and records shutdown functions so exporters flush on Close. Exporter selection is environment-driven: OTLP/HTTP when OTEL_EXPORTER_OTLP_ENDPOINT (or the signal-specific variable) is set, and a stdout exporter otherwise. Resource attributes resolve from OTEL_SERVICE_NAME, OTEL_SERVICE_VERSION, OTEL_DEPLOYMENT_ENVIRONMENT_NAME, and OTEL_RESOURCE_ATTRIBUTES.
  • metrics/statsd pushes counters, gauges, and timers over UDP or TCP to a StatsD daemon. Being push-only, its MetricsHandlerFunc returns 501 and InstrumentDB is a no-op.

Because handlers depend on the interface, swapping Prometheus for OpenTelemetry is a change at the composition root, not in the handlers. See /docs/dependency-footprint/ for what each backend costs to import.

Trace IDs

traceid captures a request-scoped correlation ID at the service boundary and carries it through, without coupling business logic to a tracing SDK.

// inbound
id := traceid.FromHTTPRequestHeader(r, traceid.DefaultHeader, traceid.DefaultValue)
ctx := traceid.NewContext(r.Context(), id)

// anywhere in the call chain
logger.With(traceid.DefaultLogKey, traceid.FromContext(ctx, traceid.DefaultValue)).Info("processing")

// outbound
traceid.SetHTTPRequestHeaderFromContext(ctx, req, traceid.DefaultHeader, traceid.DefaultValue)

FromHTTPRequestHeader validates the inbound value against [0-9A-Za-z._-], at 1 to MaxIDLen characters, and substitutes the caller-supplied default for anything else. Header injection into your logs stops there. NewContext is a no-op when an ID is already present, so it is safe to call at every layer; ForceContext overwrites when the authoritative ID has just been decided.

The conventional defaults are exported so services in one system share naming without hardcoded strings: DefaultHeader (X-Request-ID), DefaultValue, and DefaultLogKey (traceid).

httpclient does this automatically for outbound calls: when the context carries no trace ID, it generates a UUIDv7-based one and attaches it to both the context and the request headers.

Redaction

redact removes secrets from log lines and HTTP dumps before they are emitted. Every pattern is matched in a single pass over the input, so cost does not grow with the number of rules enabled.

safe := redact.Default().String(rawPayload)

One pass covers sensitive HTTP headers (preserving the name, replacing the value), JSON keys whose name tokenizes to a sensitive keyword (replacing a whole nested object or array when the value is one), URL-encoded pairs, XML elements, URL userinfo passwords in bare DSNs, JWT and JWE compact tokens, vendor credential literals by prefix (GitHub, Slack, Stripe, OpenAI and Anthropic, Hugging Face, SendGrid, AWS, Google, GitLab, and others), PEM private key blocks, and credit-card numbers.

Key matching is token-exact after normalisation, so apiKey, api_key, API-KEY, and APIKey all match while monkey never matches key and wildcard never matches card. House-style names are added with WithExtraTokens, and fields that must stay readable are exempted with WithoutTokens.

redact.New builds an independent, immutable, concurrency-safe instance. A method value such as re.BytesToString satisfies the redact-function option of httpclient, httpserver, and httpreverseproxy, which fall back to Default when the option is omitted. Redaction is never lost by omission; it is lost only by naming the bypass, InsecureNoRedaction.

Redaction has two limits worth knowing. It is best-effort pattern matching, well short of a data-loss prevention system, and it anchors on structure, so Go’s own rendering of composite values (fmt.Sprintf("%+v", req), map[password:secret]) passes through untouched, as do multipart/form-data field values, whose name and value are decoupled across lines. Marshal to JSON first, or redact those fields yourself.

Putting It Together

bootstrap wires all four: it creates the metrics client and the logger, and with WithLogConfig installs the hook that counts log records by level. httpserver logs each request with its status code and size, propagates the trace ID, and redacts HTTP data. httpclient carries the trace ID outbound and redacts query strings at every log level, and its debug-level request and response dumps are redacted and bounded by WithMaxDumpSize (1 MiB by default).

That bound has a consequence for streaming endpoints. Dumping an unknown-length response reads up to the cap before the log entry is emitted and before the body reaches the caller, which adds that much buffering latency. Keep debug logging off for server-sent events and long polling, or lower the cap.

Previous: /docs/dependency-footprint/

Overview: /docs/

Next: /docs/resilience/