filter

Your support helps keep this website running. Please consider supporting this project. SPONSOR DONATE

08 Sep 2026

Part of nurago, a collection of independent Go packages for backend services.

import "github.com/tecnickcom/nurago/pkg/filter"

Package filter provides declarative, rule-based filtering for in-memory slices. It evaluates structured Rule expressions against slice elements and filters the slice in place.

Rules are grouped as [][]Rule with boolean semantics:

outer slice: AND
inner slice: OR

So [A, [B, C], D] evaluates as A AND (B OR C) AND D. Every group must hold at least one rule, and every rule must set all three fields; an empty group, an empty rule set, or a rule missing a field is rejected as a malformed filter.

Rules can be supplied directly or parsed from JSON via Processor.ParseJSON, and query parameter payloads can be loaded with Processor.ParseURLQuery. A JSON rule object must carry the “field”, “type”, and “value” keys (an empty “field” selects the whole element; a “value” of null is a nil reference); the JSON grammar is defined by filter_schema.json. Supplying rules directly as a [][]Rule in Go is not held to that shape; a Go caller may, for example, pass an empty rule set to match every element.

Features

Comparison operators: regexp, equality/equal-fold, prefix/suffix, contains, and numeric ordering (<, <=, >, >=) with a collection/string length fallback.
Optional negation prefix (!) for every operator.
Dot-path field selection for nested struct fields. By default selectors are matched against Go field names, case-sensitively (so Address.Country); use WithFieldNameTag to match a struct tag instead (so a JSON-style address.country).
In-place filtering with pagination-style controls via Processor.ApplySubset (offset + length), plus total-match count.
Limits to constrain runtime and untrusted-input cost: rule/result counts (WithMaxRules, WithMaxResults), value length (WithMaxValueLength), payload size (WithMaxFilterBytes), and field-path depth (WithMaxFieldDepth).
Reflection-path caching for repeated evaluations on the same types.

Important Behavior

The slice argument for Processor.Apply / Processor.ApplySubset must be a pointer to a slice and is modified in place: matching elements are compacted to the front and the slice is shortened (its backing array is not zeroed beyond the new length).
A field selector that cannot be resolved against a concrete element type (it names no such field, descends into a non-struct, targets an unexported field, or is deeper than WithMaxFieldDepth) is a deterministic client error, rejected with ErrInvalidFilter before any element is touched. A field that is merely unreachable on a given element makes that element a non-match (filtered out) rather than an error; this covers a nil pointer along the path, or a field absent from the concrete type of an element in an interface-typed slice (e.g. []any), knowable only per element. A pointer or interface leaf is dereferenced to the value it holds, and a nil leaf compares as a nil operand.
Processor.ParseURLQuery returns nil rules when the configured query key is missing or empty.
The ! prefix inverts an operator’s result, including for operands the base operator cannot handle: !== against a type it cannot compare, or !< against a non-ordered operand, matches. For any element whose field is actually read, exactly one of a rule and its negation matches. They fail to partition the slice only when the field is unreachable on an element (a nil pointer along the path, or a field absent on a []any element), since such an element is a non-match for both the rule and its negation (the evaluator is never reached). != negates equal-fold (=), i.e. it is “not equal under case folding”.
A Processor and a compiled [][]Rule are safe for concurrent use across goroutines. That safety does not extend to a target slice shared between concurrent Processor.Apply calls, since Apply mutates it in place.

Security

This package performs filtering only; it does NOT perform authorization. Filter expressions are untrusted client input, and the grammar lets a client select and compare any exported field of the elements (including nested fields). Callers MUST therefore apply it only to data the requesting user is already authorized to see: pre-filter the slice to that user’s permitted records (and, where relevant, project away fields they may not read) before calling Processor.Apply. Passing records that contain fields the user is not entitled to would let a crafted filter probe those values through the match result and total-match count.

Regular-expression rules use Go’s RE2 engine, which matches in linear time with no catastrophic backtracking, so a malicious pattern cannot cause exponential-time matching. Cost is instead driven by input size, which the Processor bounds by default: a rule’s string value (including a regexp pattern) is limited to DefaultMaxValueLength bytes (WithMaxValueLength), the raw filter payload decoded by Processor.ParseJSON and Processor.ParseURLQuery is limited to DefaultMaxFilterBytes bytes (WithMaxFilterBytes), WithMaxRules caps how many rules may be applied, and WithMaxFieldDepth bounds field-selector nesting. Array and object rule values are rejected outright by the JSON parser. The number of distinct resolved field paths cached per Processor is also capped internally, so filtering a recursive element type with an unbounded variety of selectors cannot grow memory without limit. Processor.Apply still evaluates every rule against every element regardless of the requested result window, so callers should also bound the size of the slice being filtered. Decode untrusted filter JSON with Processor.ParseJSON (or Processor.ParseURLQuery): these apply the payload-size and rule-count limits.

Errors caused by a malformed or disallowed client filter are wrapped with ErrInvalidFilter. A handler processing untrusted input can test errors.Is(err, ErrInvalidFilter) to return a generic rejection (for example an HTTP 400) and log the detail server-side, rather than returning the underlying message, which may echo client input or internal type names.

Rule Encoding

The following pretty-printed JSON:

[
  [
    {
      "field": "name",
      "type": "==",
      "value": "doe"
    },
    {
      "field": "age",
      "type": "<=",
      "value": 42
    }
  ],
  [
    {
      "field": "address.country",
      "type": "regexp",
      "value": "^EN$|^FR$"
    }
  ]
]

can be represented in one line as:

[[{"field":"name","type":"==","value":"doe"},{"field":"age","type":"<=","value":42}],[{"field":"address.country","type":"regexp","value":"^EN$|^FR$"}]]

and URL-encoded as a query parameter:

filter=%5B%5B%7B%22field%22%3A%22name%22%2C%22type%22%3A%22%3D%3D%22%2C%22value%22%3A%22doe%22%7D%2C%7B%22field%22%3A%22age%22%2C%22type%22%3A%22%3C%3D%22%2C%22value%22%3A42%7D%5D%2C%5B%7B%22field%22%3A%22address.country%22%2C%22type%22%3A%22regexp%22%2C%22value%22%3A%22%5EEN%24%7C%5EFR%24%22%7D%5D%5D

The equivalent logic is:

((name==doe OR age<=42) AND (address.country match "EN" or "FR"))

These selectors (name, age, address.country) are JSON-style names, so the Processor must be built with WithFieldNameTag to resolve them against the corresponding struct tags; see the Processor.Apply example. Without it, selectors resolve against Go field names (Name, Age, Address.Country) and a JSON-style name is rejected as an unknown field selector.

Available Rule Types

Supported rule types are:

regexp : matches the value against a reference regular expression (strings only).
== : Equal to - matches exactly the reference value. Numbers compare numerically (so an int field equals a float reference of the same value); two nils are equal.
= : Equal fold - for strings, matches when they are equal under simple Unicode case-folding, a more general form of case-insensitivity (for example AB matches ab); for numbers it behaves exactly like ==.
^= : Starts with - (strings only) matches when the value begins with the reference string.
=$ : Ends with - (strings only) matches when the value ends with the reference string.
~= : Contains - (strings only) matches when the reference string is a sub-string of the value.
< : Less than - matches when the value is less than the reference.
<= : Less than or equal to - matches when the value is less than or equal the reference.
> : Greater than - matches when the value is greater than reference.
>= : Greater than or equal to - matches when the value is greater than or equal the reference.

Rule types are matched case-insensitively, so ==, REGEXP and regexp are all valid.

The ordering operators (<, <=, >, >=) require a numeric reference value (a non-numeric reference is a configuration error, ErrInvalidFilter, not a silent false). They compare numeric values directly; for strings, arrays, slices, and maps they compare the length of the value against the reference (not lexicographic order). Anything else evaluates to false.

The string operators (regexp, ^=, =$, ~=) act only on string values; a non-string reference is a configuration error, and a non-string value being tested is a non-match.

Every rule type can be prefixed with ! to negate its result. !== matches values that are not equal, !< values that are not less than the reference, and so on. Negation inverts the whole result: a ! rule also matches when the base operator could not apply at all (a type it cannot compare, or an operand with no ordering). For any element whose field is actually read, exactly one of a rule and its negation matches; they fail to partition the elements only when the field is unreachable on an element (a nil pointer along the path, or a field absent on a []any element), which is a non-match for both. != is the negation of = (equal-fold), i.e. “not equal under case folding”, not a distinct operator.

When To Use

Filter criteria arrive from an API request or a configuration file.
Rules must be composable and evaluated against arbitrary struct fields.

Example

// Simulate an encoded query passed in the http.Request of a http.Handler
encodedJSONFilter := "%5B%5B%7B%22field%22%3A%22name%22%2C%22type%22%3A%22%3D%3D%22%2C%22value%22%3A%22doe%22%7D%2C%7B%22field%22%3A%22age%22%2C%22type%22%3A%22%3C%3D%22%2C%22value%22%3A42%7D%5D%2C%5B%7B%22field%22%3A%22address.country%22%2C%22type%22%3A%22regexp%22%2C%22value%22%3A%22%5EEN%24%7C%5EFR%24%22%7D%5D%5D"

u, err := url.Parse("https://example.invalid/items?filter=" + encodedJSONFilter)
if err != nil {
	log.Fatal(err)
}

// Initialize the filter with options
// * WithFieldNameTag: to express the filter based on JSON tags and not the actual field names
f, err := filter.New(
	filter.WithFieldNameTag("json"),
)
if err != nil {
	log.Fatal(err)
}

// The filter matches the following pretty printed json:
//
//	[
//	  [
//	    {
//	      "field": "name",
//	      "type": "==",
//	      "value": "doe"
//	    },
//	    {
//	      "field": "age",
//	      "type": "<=",
//	      "value": 42
//	    }
//	  ],
//	  [
//	    {
//	      "field": "address.country",
//	      "type": "regexp",
//	      "value": "^EN$|^FR$"
//	    }
//	  ]
//	]
//
// can be represented in one line as:
//
//	[[{"field":"name","type":"==","value":"doe"},{"field":"age","type":"<=","value":42}],[{"field":"address.country","type":"regexp","value":"^EN$|^FR$"}]]
//
// and URL-encoded as a query parameter:
//
//	filter=%5B%5B%7B%22field%22%3A%22name%22%2C%22type%22%3A%22%3D%3D%22%2C%22value%22%3A%22doe%22%7D%2C%7B%22field%22%3A%22age%22%2C%22type%22%3A%22%3C%3D%22%2C%22value%22%3A42%7D%5D%2C%5B%7B%22field%22%3A%22address.country%22%2C%22type%22%3A%22regexp%22%2C%22value%22%3A%22%5EEN%24%7C%5EFR%24%22%7D%5D%5D
//
// the equivalent logic is:
//
//	((name==doe OR age<=42) AND (address.country match "EN" or "FR"))
rules, err := f.ParseURLQuery(u.Query())
if err != nil {
	log.Fatal(err)
}

// Given this list, the last item will be filtered
list := []ID{
	{
		Name: "doe",
		Age:  55,
		Addr: Address{
			Country: "EN",
		},
	},
	{
		Name: "dupont",
		Age:  42,
		Addr: Address{
			Country: "FR",
		},
	},
	{
		Name: "doe",
		Age:  41,
		Addr: Address{
			Country: "US",
		},
	},
}

// Filters the list in place
sliceLen, totalMatches, err := f.Apply(rules, &list)
if err != nil {
	log.Fatal(err)
}

fmt.Println(sliceLen)
fmt.Println(totalMatches)

for _, id := range list {
	fmt.Println(id)
}

// Output:
// 2
// 2
// {doe 55 {EN}}
// {dupont 42 {FR}}

Full source is in example_filter_test.go. More runnable examples are on pkg.go.dev.

Dependencies

This package reaches no external module: it uses only the Go standard library.

Table of contents