Field collapsing #

Collapse the result list so each distinct value of a field appears at most once — “best product per merchant”, “best article per author” — without a terms aggregation, keeping full hit semantics (scores, source, highlight).

POST /my-collection/_search

How it works #

Collapse is engine-native and runs as a post-retrieval step. The collector normally stops at from + size, but collapse pages over GROUPS, so the engine first gathers a wider window — max(1000, from + size) hits (COLLAPSE_WINDOW in lib/engine/src/search/mod.rs) — in request sort order, then:

1. Groups the hits by the collapse field’s value, preserving first-seen order. Documents missing the field form their own group.
2. Keeps the FIRST hit of each group as its representative — with no sort that is the highest-scoring member; with a sort it is the first under that order.
3. Pages from/size over the representatives (groups, not raw docs).
4. Leaves hits.total at the FULL match count — collapse deduplicates the result list, never the count.

inner_hits asks for each group’s runner-ups. The engine gives every inner-hits request its own sort clause (numeric compare first, text fallback, missing values always last regardless of direction, doc-id ascending as the tie-break), an optional second-level collapse that deduplicates the runner-ups, and a total counted before size truncation. Default inner-hits name is inner_hits, default size 3; the wire form accepts a single object or an array.

Delivery status of inner_hits today: the engine assembles the blocks, but the distributed transport currently drops them at the RPC boundary (collapse_inner_hits is a named TODO in the transport conversion), and the in-page widget executes the wasm engine, which also collapses without runner-ups — over the HTTP API, treat inner_hits as pending.

Examples #

One best-selling hit per product type (the dataset has four: hat, t-shirt, mug, sticker); the plain match-all total stays 120 — collapse deduplicates the RESULT list, not the count.

Collapse composes with sort: sort by price descending and collapse by type, and each representative is that type’s most expensive sale — the group order follows the same sort.

Use cases #

• One hit per owner. Deduplicate search results per user, per host, or per merchant so a single noisy owner cannot fill the first page — classic “best X per Y” result lists.
• Grouped browsing. Collapse logs by host (or level) to see one representative log per group while hits.total still reports how many logs matched overall.
• Cheaper distinct lists. A small distinct-value list with full hit bodies (source, highlights) — what a terms aggregation cannot give you, since buckets carry counts, not documents.
• Two-level grouping shape. Outer collapse for the group, inner collapse inside inner_hits for a secondary key when rendering grouped result cards (conformance-suite path today).

Parameters for collapse #

• field
  (Required, string) The field to collapse on. Keyword-like fields with doc values give the cheapest collapse.
• inner_hits
  (Optional, object | array of objects) Also return the top runner-up documents of each group:
  • name (Optional, string, default: inner_hits) Key under which the inner hits appear.
  • size (Optional, integer, default: 3) Runner-ups per group.
  • sort (Optional, array) Sort clause for the runner-ups; equal keys tie-break by internal doc id ascending.
  • collapse (Optional, object) Nested collapse for a second-level grouping of the runner-ups.
• max_concurrent_group_searches
  (Optional, integer) Accepted for wire compatibility; the engine’s group assembly is not parallelized today, so the value is unused.

Notes #

• Collapse runs after scoring, so the top hit per group is the best match, not the newest document (unless you sort by date).
• Pagination applies to collapsed groups, not raw documents.
• rescore and collapse in one request: the ES-compat layer rejects the mix; the native gateway path currently executes both — do not rely on that combination.
• For full per-group analytics (counts, aggregations over group members) use a terms aggregation with sub-aggs instead; collapse is for result-list shape.

Related #

• Sort and pagination — the other half of result-list shape: which group leads, and where missing values land.
• Rescore — re-rank the window before (or instead of) collapsing; the third member of the result-shaping family.
• terms aggregation — per-group counts and sub-aggregations when you need analytics, not hits.
• Hybrid search — retrieve (hybrid) requests cannot carry a collapse block; the gateway rejects the mix.