prometheusgrpc

prometheusgrpc collects Prometheus metrics for gRPC servers and clients, with the upstream metric names and one registry shared with the HTTP metrics.

-

Table of contents

Part of nurago, a collection of independent Go packages for backend services.

import "github.com/tecnickcom/nurago/pkg/metrics/prometheus/prometheusgrpc"

Prometheus metrics for grpcserver and grpcclient, built on the go-grpc-middleware provider.

m, err := prometheusgrpc.New()

client, err := prometheus.New(prometheus.WithCollector(m))

grpcserver.WithInstrumenter(m)
grpcclient.WithInstrumenter(m)

One registry, one endpoint

A Metrics value is a prometheus.Collector, so it registers with the WithCollector option of prometheus, next to the HTTP and runtime metrics. The existing /metrics route then serves both, and the scrape configuration does not change when gRPC is added.

The same value implements both instrumenter interfaces from grpcutil, so one instance covers the server and the client.

Upstream names, so dashboards work

The metric names are the go-grpc-middleware ones:

MetricLabels
grpc_server_started_total, grpc_client_started_totalgrpc_type, grpc_service, grpc_method
grpc_server_handled_total, grpc_client_handled_totalthe above plus grpc_code
grpc_server_msg_received_total, grpc_server_msg_sent_total, and the client equivalentsgrpc_type, grpc_service, grpc_method
grpc_server_handling_seconds, grpc_client_handling_secondsgrpc_type, grpc_service, grpc_method

Published Grafana dashboards and alert rules for gRPC are written against these names, and they work without edits.

The handling-time histograms are on by default, with the buckets of the HTTP duration histograms in prometheus (0.001 to 100 seconds), so gRPC and HTTP latencies are comparable on one panel. WithServerHandlingTimeBuckets and WithClientHandlingTimeBuckets change them, and WithoutHandlingTimeHistogram turns them off.

Series exist before the first call

When the server starts, every registered method gets its series at zero. A rate or ratio over a method that has never been called is then zero rather than absent, and an alert on error ratio does not stay silent until the first error happens to create the series.

Health and reflection are filtered out

By default grpc.health.v1.Health and grpc.reflection.* RPCs are neither recorded nor pre-created. A load balancer probing every few seconds would otherwise dominate the request rate. WithFilter adds filters; WithoutDefaultFilter removes the default one.

WithExemplarFromContext(prometheusgrpc.TraceIDExemplar) attaches the trace ID that grpcserver and grpcclient store in the context to every observation, so a latency outlier on a dashboard leads to the log lines of that RPC. Exemplars appear only in the OpenMetrics format: set EnableOpenMetrics in the promhttp.HandlerOpts passed with prometheus.WithHandlerOpts.

Dependencies

Reaches github.com/prometheus/client_golang, the go-grpc-middleware Prometheus provider, and google.golang.org/grpc: 16 external modules in total.