kafka

kafka produces and consumes Apache Kafka messages in pure Go, with no CGO and no librdkafka, and explicit at-most-once and at-least-once consumer paths.

-
Updated:

Table of contents

Part of nurago, a collection of independent Go packages for backend services.

import "github.com/tecnickcom/nurago/pkg/kafka"

Producer and consumer for Apache Kafka, built on segmentio/kafka-go.

p, err := kafka.NewProducer(urls, topic)
err = p.SendData(ctx, key, payload)

c, err := kafka.NewConsumer(urls, topic, groupID)
msg, err := c.Receive(ctx)

Pure Go, no CGO

No librdkafka to install, no CGO, so CGO_ENABLED=0 still builds.

The practical consequences are all about deployment. A statically linked binary runs in a scratch or distroless container with nothing else in it. Cross-compiling for another platform is GOOS=linux GOARCH=arm64 go build rather than a toolchain and a C library for the target. And go test needs no system package installed on the machine or in CI.

The confd-based alternative is faster at very high throughput. For the volumes most services handle, not needing a C toolchain in every build image is worth more.

Delivery semantics are chosen by which method you call

This is the part to get right, and the package makes the choice explicit rather than burying it in configuration.

Producer: writes wait for acknowledgement from the full in-sync replica set by default (kafka.RequireAll). That is the safe setting: a message is not considered sent until every in-sync replica has it, so a broker failure immediately afterwards does not lose it. WithRequiredAcks relaxes it for throughput, at the cost of a window where an acknowledged message can be lost.

Consumer, with a group configured:

  • Receive and ReceiveData are at-most-once. The offset is committed as part of reading, so a crash during processing loses the message.
  • FetchMessage plus CommitMessages is at-least-once. You commit after processing succeeds, so a crash means the message is redelivered.

At-least-once is what most work needs, and it requires idempotent handling, since redelivery is a normal event rather than an error. stringkey produces a stable idempotency key from message fields.

Reaching for Receive because it is the shorter call is how messages get lost quietly.

Options

WithSessionTimeout bounds how long the group coordinator waits before deciding a consumer is gone and rebalancing. Too short and a slow consumer triggers rebalances; too long and a dead consumer’s partitions stay unassigned.

WithStartOffset sets where a new consumer group begins: the earliest retained message, or only new ones. Replaying from the earliest is the right choice for a rebuild and the wrong one for a notification consumer that would re-send everything.

Batching options tune producer throughput against latency.

Custom codecs replace the default encode and decode, so a message body can be Protobuf, Avro, or whatever your topic contract specifies rather than the default encoding.

Dependencies

Reaches github.com/segmentio/kafka-go.