strsplit

Part of nurago, a collection of independent Go packages for backend services.

import "github.com/tecnickcom/nurago/pkg/strsplit"

ChunkLine splits one line into pieces of at most size bytes. Chunk splits a block of text, treating newlines as the first-choice boundary and handing long lines to ChunkLine. Both take an optional chunk limit.

The use case is anything with a hard payload ceiling: an SMS segment, a log field, a message body, a column width, an embedding window.

Bytes, not runes, and the one exception

size is a maximum byte length. That is the right unit, because the limits this exists to respect are byte limits, and a rune count tells you nothing about whether a UTF-8 payload fits in 160 bytes.

Splitting on a byte boundary would produce invalid UTF-8, so the cut is always moved back to a rune boundary. Which leaves one case the package is honest about: a single rune wider than size is emitted whole, and that one chunk exceeds the limit. There is no alternative that produces valid UTF-8. If your ceiling is absolute, check the result rather than trusting the parameter.

The separator preference

Within the size window, ChunkLine looks backwards for a place to cut, in this order:

1. Unicode whitespace.
2. Unicode punctuation, kept with the preceding chunk.
3. A hard rune-safe cut, if neither exists.

Punctuation staying with the chunk before it is the detail that makes the output readable: a chunk ending in sentence. reads as finished, where one beginning with .Next does not.

What Chunk does to your whitespace

Chunk is opinionated in ways that will surprise you if you expected a faithful splitter:

• Only \n is a line boundary. A bare \r is not, and neither are U+2028, U+2029 or U+0085. Those can remain inside a chunk.
• \r is removed only where it sits next to a \n, as a side effect of trimming each line.
• Every line is whitespace-trimmed.
• Empty and whitespace-only lines are dropped, so blank lines between paragraphs do not survive.

ChunkLine does none of this. It takes the string exactly as given and only applies the size and separator rules, so calling it against your own line splitting preserves whatever structure Chunk would have normalised away.

The chunk limit

Both functions take n, a maximum number of chunks. Pass -1 for unlimited. Pass 0 and both return nil.

A positive n stops after that many chunks, so the tail of a long input is dropped rather than split further. That makes ChunkLine(s, size, 1) a bounded-prefix operation: at most size bytes, cut at a rune boundary and preferring a separator.

Text without separators costs about five times more per byte

Measured with go test -bench on Go 1.27.1, linux/amd64, Intel Core i7-1260P. Run go test -bench=. ./pkg/strsplit/ to reproduce them on your own hardware; the absolute values will differ, the ratios much less so.

The prose and no-separator rows are not the same input size, so compare per byte: roughly 1.1 ns/byte for prose against 5.5 ns/byte for an unbroken run of characters.

The reason is in separatorEnd. It scans backwards through the window for whitespace, and if that fails it scans backwards again for punctuation. On prose the first scan usually succeeds within a few bytes. On a 2,000-character string with no spaces, both scans traverse the entire window and both fail, for every chunk, before the hard cut happens anyway.

Worth knowing before you chunk base64, a hex dump, or a long identifier: the pathological input for this function is exactly the input with no natural break, and it is five times the cost with none of the readability benefit.

ChunkMultiline is the allocation-heavy one at 43 allocations, because each line produces its own result before the chunks are gathered.

Example

chunks := strsplit.ChunkLine("the quick brown fox jumps over the lazy dog", 12, -1)

fmt.Println(chunks)

Full source is in example_strsplit_test.go. More runnable examples are on pkg.go.dev.

Dependencies

This package reaches no external module: it uses only the Go standard library.

Links

• Reference documentation
• Source code
• All nurago packages