Resilience

Every dependency a service calls will fail sometimes. These packages decide when to try again and how long to wait. They share one delay calculator, so a single retry policy applies to HTTP and non-HTTP work alike.

The Delay Schedule

backoff is a pure calculator: no timing, no sleeping, no goroutines, no I/O. Given a base delay, a growth factor, a jitter ceiling, and a maximum, each call to Schedule.Next returns the next delay and advances the progression. You own the loop and the timer.

s := backoff.New(backoff.Config{
    Base:     100 * time.Millisecond,
    Factor:   2,
    Jitter:   50 * time.Millisecond,
    MaxDelay: 30 * time.Second,
})

for {
    // ... attempt the work ...
    time.Sleep(s.Next()) // 100ms, 200ms, 400ms, ... capped at 30s, each with jitter
}

Three jitter strategies are selectable through Config.Strategy. The default, JitterAdditive, adds a random amount in [0, Jitter); because the ceiling is fixed, its desynchronising effect shrinks in relative terms as the delay grows. JitterFull replaces the delay with a uniform random value in [0, delay), which decorrelates concurrent clients best because the spread scales with the delay. JitterEqual keeps half the delay and randomises the other half, giving a wait in [delay/2, delay): less spread, but a floor on how eagerly you retry.

Both the internal progression and the jitter addition are clamped, so no delay can overflow into a negative duration regardless of factor, attempt count, or configured maximum.

The overflow itself: a time.Duration accumulator that keeps doubling passes math.MaxInt64 and wraps negative. A guard written as if delay > maxDelay lets the negative value through, since it is not greater than anything positive. time.Sleep reads it as “return immediately”. The retry loop then runs flat out against a dependency that was already failing, at the attempt count where backoff was supposed to matter most. Capping the returned delay does nothing here, because only the internal state overflows.

AddJitter exposes the jitter step alone, for code that paces work at a fixed interval rather than backing off.

Retrying a Task

retrier executes a task function with that schedule, a retry predicate, and a per-attempt timeout.

r, err := retrier.New(
    retrier.WithAttempts(5),
    retrier.WithDelay(200*time.Millisecond),
    retrier.WithDelayFactor(2),
    retrier.WithJitter(25*time.Millisecond),
)
if err != nil {
    return err
}

err = r.Run(ctx, func(ctx context.Context) error {
    return callExternalService(ctx)
})

Each attempt runs under its own timeout context, the result is evaluated by a RetryIfFn, and the loop stops when attempts are exhausted or the predicate says no. Parent context cancellation is always respected. The defaults are 4 attempts, 1s initial delay, factor 2, 1ms jitter, a 1s per-attempt timeout, an unbounded maximum delay (cap it with WithMaxDelay), and retry on any non-nil error.

The per-attempt timeout is separate from the overall deadline, so one hung attempt cannot consume the whole budget.

Retrying an HTTP Request

httpretrier wraps an existing HTTP client rather than replacing it, so a client that is already instrumented or authenticated keeps its behaviour. Retries are configured per request, not per client.

The retry predicate receives both the *http.Response and the error, so policy can key on transport failures and status codes together. Three built-in policies cover the common cases: RetryIfForReadRequests for idempotent reads, RetryIfForWriteRequests for state-changing writes, and RetryIfFnByHTTPMethod to select between them from the method name. The default retries only on a transport error.

WithRespectRetryAfter makes the retrier wait at least the server-provided Retry-After delay, routed through the same jitter helper as every other delay in the library and still bounded by your maximum.

Replaying a request with a body relies on Request.GetBody. When the body cannot be recreated, Do returns an error instead of retrying with a consumed reader. Bodyless requests retry without restriction.

WithOnRetry surfaces each scheduled retry for logging or metrics.

Periodic Work

periodic runs a task at a fixed interval with optional jitter and a per-invocation timeout.

p, err := periodic.New(
    30*time.Second, // interval
    5*time.Second,  // jitter ceiling
    10*time.Second, // per-call deadline
    myTask,
)
if err != nil {
    return err
}

p.Start(ctx)
defer p.Stop()

The pause between calls is interval + rand(0, jitter), which spreads steady-state load across a fleet. By default the first call fires immediately rather than after a full interval, and that first call is not jittered: a fleet that starts in lockstep (a rolling deploy, a simultaneous restart) fires every replica’s first call together. Pass WithInitialJitter to spread it too.

Cancelling the parent context or calling Stop ends the loop after the current invocation returns, without leaking the goroutine. The task must not panic: it runs in a background goroutine with no recovery, so recover inside the task if it can.

DNS Caching

dnscache is a concurrency-safe, size-bounded DNS cache with single-flight collapsing, exposing LookupHost and a DialContext that drops into an http.Transport. Resolved names are held for one cache-wide TTL; the authoritative record TTLs are not consulted. Concurrent callers asking for the same host share one lookup, and host names are matched case-insensitively with the trailing-dot FQDN form treated as equivalent.

It reaches no external module.

Single-Flight Caching

sfcache covers the neighbouring problem: an expensive lookup that many goroutines want at once. It is a bounded, thread-safe cache with TTL and single-flight deduplication, so a cache miss under load produces one upstream call rather than one per caller. awssecretcache applies the same model to AWS Secrets Manager lookups.

Choosing Between Them

• An existing retry loop that needs only the delays: backoff.
• Arbitrary work against a flaky dependency: retrier.
• Outbound HTTP, where status codes and Retry-After matter and bodies must be replayed: httpretrier.
• Work on a schedule: periodic.

Whichever you pick, retries multiply load on a dependency that is already failing, so bound the attempts and jitter the delays. Where many clients back off together, full jitter spreads them furthest.

Previous: /docs/observability/

Overview: /docs/

Next: /docs/security/