stringkey

Part of nurago, a collection of independent Go packages for backend services.

import "github.com/tecnickcom/nurago/pkg/stringkey"

Takes several text fields and produces one uint64 that is the same every time for inputs that differ only in ways you do not care about.

sk := stringkey.New("  John  Doe ", "LONDON")
sk.Key()     // uint64
sk.Hex()     // fixed 16-character hex
sk.String()  // base-36, shorter and variable length

The pipeline is the product

The hash is the boring part. What makes this a package rather than a call to a hash function is the six-step normalisation in front of it, applied identically every time:

1. Trim leading and trailing Unicode whitespace, per field.
2. Collapse runs of Unicode whitespace to a single space.
3. Lowercase.
4. Join fields with a tab.
5. Apply Unicode NFC normalisation.
6. Hash with FarmHash64.

Step 4 prevents a specific collision. Concatenating fields without a separator maps ("ab", "c") and ("a", "bc") to the same bytes, and any separator that can occur inside the data reintroduces the problem. A tab survives step 1 as a boundary and is vanishingly rare inside a name or an address field.

Two normalisation limits, documented

NFC, not NFKC. Canonically equivalent forms collapse, so a precomposed é and a decomposed e plus combining accent produce the same key. Compatibility variants do not collapse: the ff ligature stays distinct from ff, full-width forms stay distinct from half-width, superscripts stay distinct from their base characters.

NFKC would additionally fold ² into 2, altering meaning rather than representation.

Locale-independent lowercasing. It uses unicode.ToLower, the simple mapping, rather than language-tailored case folding. The same input therefore produces the same key on every machine regardless of locale settings, so a stored key stays portable between services. Turkish dotless ı and dotted İ follow the default mapping rather than Turkish rules, so I and ı do not fold together.

Stability is a promise, with one footnote

For a given input the key does not change across runs, architectures, or Go versions. FarmHash64 is a fixed algorithm and the output encodings are plain integer formatting. That is what makes the key safe to persist, to use as a primary key, or to send to another service and compare.

The footnote is step 5. Unicode normalisation uses the data tables from golang.org/x/text, and those tables gain entries as Unicode assigns new code points. ASCII input and long-assigned characters are effectively permanent. Only rare or newly-assigned code points could normalise differently after a dependency upgrade.

For most datasets that risk is theoretical. If yours stores keys forever and accepts arbitrary Unicode, pin the dependency and treat an upgrade as a migration.

Cost depends on the text, not on the API

There is one constructor. What changes is what you hand it:

Measured with go test -bench on Go 1.27.1, linux/amd64, Intel Core i7-1260P. Run go test -bench=. ./pkg/stringkey/ to reproduce them on your own hardware; the absolute values will differ, the ratios much less so.

Both allocate the same 72 bytes, so the 3.7x gap is time spent in the Unicode work rather than in memory: case mapping and NFC normalisation over CJK and combining-diacritic input, plus the whitespace collapsing that the second input’s padded field triggers.

The practical reading is that the key costs well under a microsecond either way. Two allocations and a few hundred nanoseconds is cheap enough to call on a per-record path while loading a file, and far cheaper than the database round trip the key is usually there to avoid.

Collision properties

FarmHash64 is a non-cryptographic hash, chosen for throughput and distribution quality rather than for resistance to a chosen-input attack.

The 64-bit output width sets the practical ceiling. By the birthday bound, the chance of at least one collision passes 50% at roughly 5 billion distinct keys, and is around one in 10 million at 6 million keys. For a deduplication or idempotency workload sized in the millions, that is comfortable; for one sized in the billions, store the full input alongside the key and compare on collision.

Cryptographic hashing and authenticated encryption are in encrypt, and password-specific hashing in passwordhash.

Example

sk := stringkey.New("John", "Doe")

fmt.Println(sk.Hex())

Full source is in example_stringkey_test.go. More runnable examples are on pkg.go.dev.

Dependencies

Reaches golang.org/x/text for Unicode normalisation.

Links

• Reference documentation
• Source code
• Fuzzy comparison instead of exact keys: stringmetric
• All nurago packages