Table of contents
Services leak secrets into their own logs all the time. Someone dumps an inbound HTTP request to debug a flaky integration and the Authorization header goes straight to disk. A JSON body with a password field lands in an error log. A connection string with the database password ends up in a stack trace. Nobody did anything reckless. They logged what they needed to see when something broke, and the credentials came along for the ride.
The redact package in nurago works at that boundary, in the moment before text reaches the log, and strips the sensitive parts out.
Where it runs is what shaped it. It is the fallback redact function for the request and response dumps of nurago’s httpclient, httpserver, and httpreverseproxy packages, and it is meant to run on every log line. Every byte of every line goes through it. That sets a budget: under a microsecond for a typical line, allocating nothing, or someone will switch it off.
An obvious implementation is a pile of regular expressions, one per shape, each one a full scan of the string. Backtracking makes that degrade badly on adversarial input, which is exactly what a log redactor tends to be fed.
One table load per byte
redact’s engine makes a single pass. For each byte it consults a 256-entry byte-class table (bulkTrigger in the source) that answers one question: could a redaction rule possibly start here? For almost every byte of a log line the answer is no, and the scan costs one table load before moving on. When the run of safe bytes ends, the whole run is appended to the output in a single bulk copy.
Remaining byte classes are filtered before any rule runs. Hard stops (", =, <, newline, digits) always hand over to the rules. Candidate bytes get a short inline prefilter first. A : only stops the scan when // follows, meaning a URL that may carry userinfo credentials. An e only counts as eyJ at a word boundary, the opening of every JSON Web Token. A - only as the -----B of a PEM boundary. The nine first letters of vendor token prefixes only when the three bytes starting there spell a known prefix: ghp, sk-, AKI, and the rest. Prose containing “skip” or “energy” never leaves the bulk-copy loop.
Digits get their own fast path ahead of rule dispatch, because identifier-heavy log lines are dominated by digit runs: trace IDs, UUIDs, ports, durations. A run glued to word characters is copied verbatim as part of an identifier. Only a free-standing run is checked as a candidate card number.
What survives all of that reaches per-rule dispatch, where each trigger byte maps to one rule class: sensitive HTTP headers and JSON keys, key=value pairs in URL-encoded form data, XML elements, URL userinfo passwords, JWT and JSON Web Encryption (JWE) compact tokens, vendor credential literals, Privacy-Enhanced Mail (PEM) private-key blocks, and card numbers. Any class can be switched off per instance with WithoutRules.
Lookahead is bounded as well. A rule that inspects a candidate and rejects it does not consume input, so an unbounded forward search could be re-entered at every trigger byte. Feed it a line packed with -----BEGIN markers or unterminated XML comments and redaction turns quadratic: a denial of service (DoS) in the logging path. The engine caps those searches, the inline PEM end-marker search at a 16 KB window and the XML comment and CDATA terminator search at 8 KB, so total work stays linear. The fallbacks then consume what they scanned, erring in the safe direction: an unterminated private-key body is redacted through to the end of its value rather than left visible.
Keyword matching without allocating
Deciding whether a key name like X-Api-Key or dbPassword2 is sensitive happens thousands of times a second, so it cannot allocate. The matcher walks the key’s tokens in place, splitting on camelCase, snake_case, kebab-case, and acronym-run boundaries as it goes. Each token is lowercased into a fixed stack buffer and looked up through Go’s allocation-free map[string(bytes)] optimisation, which keeps the keyword set an ordinary map without paying for a string conversion per token. Keys containing non-ASCII bytes take a slower normalising path, and those results are memoised in a bounded per-instance cache.
Output can be allocation-free too. AppendTo writes into a caller-owned buffer, while Pooled and BytesToString draw their scratch buffer from a sync.Pool.
re := redact.Default()
var dst []byte
for _, payload := range payloads {
dst = re.AppendTo(dst, payload)
logger.Info("request", "payload", string(dst))
}
The numbers, from make bench in the repository:
$ go test -run='^$' -bench=. -benchmem ./pkg/redact/
goos: linux
goarch: amd64
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
BenchmarkString-16 1000000 1040 ns/op 408 B/op 3 allocs/op
BenchmarkBytes-16 1327519 911 ns/op 208 B/op 1 allocs/op
BenchmarkAppendTo-16 1422664 830 ns/op 0 B/op 0 allocs/op
BenchmarkPooled-16 1369544 875 ns/op 24 B/op 1 allocs/op
BenchmarkAppendToDigitHeavy-16 1290297 940 ns/op 0 B/op 0 allocs/op
Those first four run against a 206-byte HTTP request dump carrying four separate secrets, which is close to the worst case for rule dispatch: roughly 4 ns per byte, and AppendTo does it without allocating. The last one is a 173-byte structured log line of trace IDs, a UUID, a host and port, and a duration, with nothing in it to redact at all. It is slower per byte than the dump, because proving a long digit run uninteresting is the most expensive negative result the engine produces.
The correctness layer
None of that matters if the output cannot be trusted, and it can fail in two directions: a leaked secret, or a log line shredded into a wall of markers.
Boundary-exact key matching
Keyword matching is token-exact rather than a substring search. apiKey, api_key, API-KEY, and APIKey all tokenise to api + key and match. monkey does not match key, and wildcard does not match card.
Around the exact match sit a few bounded generalisations. A trailing digit run is stripped, so password2 and cvv2 match. A trailing plural s is retried against a short list of unambiguous roots only: tokens redacts, while keys, a JSON Web Key Set array, stays visible. All-lowercase glued compounds match when they end in one of those roots, which catches newpassword and awssecretkey. A few two-word pairs match together where neither word is sensitive alone: firstName, nationalId, connectionString.
House-style names go in per instance with WithExtraTokens. Over-eager ones come out with WithoutTokens, which is how you keep amount and balance readable in fintech logs.
Convergent output
In real systems the same string often passes through more than one logging layer, so redacting already-redacted text has to be safe. The property the package holds: re-redacting output never reveals more than the first pass did, output is byte-stable in a single pass on well-formed input, and on structurally ambiguous input it reaches a fixed point after at most one extra pass.
Rules are written to preserve it. The marker is inert text, and redaction does not consume structural bytes that would change how a second pass parses the surroundings. A dedicated fuzzer and a set of regression tests pin the property down.
Cards: over-redact by default, verify on request
Card detection over-redacts on purpose. Any free-standing run of 13 to 19 digits, contiguous or grouped by single spaces or dashes, that matches a known network prefix and length gets masked, even when that catches unrelated identifiers of the same shape. Letting a real Primary Account Number (PAN) through is the worse failure of the two. Grouped detection excludes a few legacy ranges whose prefixes collide with phone-number formats, such as 1 800 555 0199 1234.
Callers who prefer fewer false positives can turn on a Luhn-checksum gate, which then demands both a prefix match and a valid checksum. As a side effect it unlocks detection of short 12 to 15 digit Maestro numbers, too collision-prone to match on prefix alone. The gate is fixed per instance at construction rather than exposed as a process-global toggle, so one component flipping it cannot silently change what every other component logs:
re := redact.New(redact.WithLuhnCheck(true))
safe := re.String(rawPayload)
One entry point, one named bypass
All redaction runs through a Redactor, immutable after construction and safe for concurrent use. redact.Default() returns the shared zero-configuration instance: marker ***, all rules on, Luhn gate off. redact.New builds an independent one.
import "github.com/tecnickcom/nurago/pkg/redact"
safe := redact.Default().String(rawLogLine)
re := redact.New(
redact.WithMarker("#REDACTED#"), // custom placeholder
redact.WithExtraTokens("floof"), // company-specific key names
redact.WithoutTokens("amount", "balance"), // keep fintech fields readable
redact.WithoutRules(redact.RuleCards), // or disable a whole rule class
)
safe = re.String(rawPayload)
Disabling redaction outright is possible, but only in the open. redact.InsecureNoRedaction is a ready-made pass-through for the redact-function options of the HTTP packages, named after the crypto/tls.InsecureSkipVerify convention so that it stands out in review. An unset option falls back to Default() and a nil function is ignored, so redaction is never lost by omission, only by writing that name into a diff.
Adding it to a service
pkg/redact imports the standard library and nothing else. From a clean module that imports only this package:
$ go list -deps -f '{{if .Module}}{{.Module.Path}}{{end}}' . | sort -u
example.com/probe
github.com/tecnickcom/nurago
$ wc -l go.sum
6 go.sum
One require line, no indirect entries, two modules in the build. The 70-package monorepo does not follow the import.
Wiring it into the HTTP packages is a method value, since Redactor.BytesToString already has the func([]byte) string shape their options expect:
re := redact.New(redact.WithoutTokens("amount", "balance"))
cli := httpclient.New(httpclient.WithRedactFn(re.BytesToString))
Anywhere else, redact.Default().String(s) on the way into the logger covers the common case, and AppendTo with a reused buffer covers the hot path.
Where it fits, and where it stops
redact is not a Data Loss Prevention product. It does not classify data across an organisation or police every channel data can leave through. It solves the smaller problem of a fast, predictable sanitisation step at the boundary where a service turns internal state into text that persists. It pairs with structured logging, by routing handler or HTTP dump output through it, and with the jwt package, whose default responder logs issued tokens at debug level unless it is handed a redacting logger.
Scope limitation is structural. Redaction is pattern-based, so it only catches shapes it can anchor on. A bespoke credential format it was never told about passes through, and WithExtraTokens is the way to name one. So does anything with no structure to anchor on. Go’s %+v rendering of a struct or map has none of the quoted-key, key=value, or header shapes. A multipart form body separates each field name from its value across lines. Both have to be redacted field by field, or marshalled to JSON first.
So it is a safety net under disciplined logging, and it should be budgeted as one. What it buys is that the recurring “we leaked a token in the logs again” incident becomes rare enough to be surprising.