Diversified sampler aggregation #

Like sampler, but the sample is diversified by a field — up to one (or few) documents per value — so rare categories survive the sampling. A plain sampler can be flooded by the most common value (every sampled type being t-shirt); the diversified variant guarantees representation for every distinct value first.

How it works #

The sampler walks the matching documents in two passes. First it caps each distinct value of the diversify field at max_docs_per_value (default 1) — extra documents with an already-full value are dropped. Documents missing the diversify field share one common group. The survivors are then sorted and truncated to the overall shard_size (default 100).

The result is a single bucket keyed sample carrying the sample’s doc_count and the sub-aggregation results — the same response shape as sampler; only the composition of the sample differs.

The two knobs trade reach for balance: max_docs_per_value decides how deeply each value may be represented, and shard_size bounds the total sample. With max_docs_per_value at 1, the sample holds at most one document per distinct value (a “one of each” specimen collection).

For each value the FIRST matching documents are the ones kept — the cap fills in candidate order — so, like sampler, the choice is deterministic rather than random, but no single value can crowd out the rest before the shard_size cut is applied.

Example #

Sample with at most one document per product type: all four types appear instead of being dominated by t-shirt:

Use cases #

• Fair category coverage. The teaching products split 8 / 8 / 7 / 7 across four categories; a diversified sample guarantees every category is represented even at small shard_size.
• Undominated term counts. A terms sub-aggregation over a diversified sample of sales.type keeps sticker and mug visible next to the 60 t-shirt sales instead of drowning in them.
• Balanced significance mining. Diversify by author, host, or merchant before a significant_terms pass, so one prolific source cannot dominate the foreground.
• Duplicate-adjacent data. When documents arrive with copy-pasted payloads (retries, mirrored sensors), capping per source value keeps the sample from collapsing onto the noisiest emitter.

Parameters for diversified_sampler #

• field
  (Required, string) Field used to diversify the sample.
• shard_size
  (Optional, integer, default: 100) Sample size per shard.
• max_docs_per_value
  (Optional, integer, default: 1) Documents kept per distinct value.
• aggs
  (Optional, object) Sub-aggregations computed over the sample.

Related #

• sampler — the undiversified version
• terms — the typical sub-aggregation run over a sample
• significant_terms — pattern partner: sample, then rank