Extended stats aggregation #

A multi-value metrics aggregation extending stats with squares, variance, standard deviation and their lower/upper bounds. Where stats says what the values are, extended_stats says how tightly they cluster around the mean — the input for outlier bands, quality gates and anything else built on “how unusual is this value”.

How it works #

The engine walks the matching documents once, maintaining a running count, sum, sum_of_squares, minimum and maximum — the same single pass as stats, plus one multiply-add for the squares. Everything downstream is derived from these raw moments, and is exact (no sampling, no sketches):

• avg = sum / count
• variance (variance_population) = sum_of_squares / count − avg², clamped at zero to guard against floating-point cancellation
• variance_sampling = (sum_of_squares − sum² / count) / (count − 1) — 0 when only one value was seen
• std_deviation (and the _population / _sampling variants) = the square roots of the variances above

The population statistics divide by N and describe the matched set itself; the sampling variants divide by N−1 and estimate the underlying distribution — on large matches the two converge.

std_deviation_bounds are avg ± sigma × std_deviation, with sigma defaulting to 2 (the classic 95% band).

Documents missing the field are skipped by default; with missing set, each such document contributes the default to every accumulator. On the column-store fast path (the production executor) each segment’s column is decoded once and walked with for_each_f64. Distributed requests stay exact: per-shard count, sum and sum_of_squares merge by addition, and every variance and deviation is recomputed from the merged moments.

Example #

Returns count, min, max, avg, sum, sum_of_squares, variance, variance_population, variance_sampling, std_deviation, std_deviation_population, std_deviation_sampling plus std_deviation_bounds (upper/lower at ±2σ).

Use cases #

• Outlier banding. Over the teaching sales, avg = 27.5 and std_deviation ≈ 14.79, so the default bounds are ≈ −2.08..57.08 — the 60-priced stickers are the only sales outside the 2σ band.
• Spread per segment. Nested under terms, each product type prices uniformly in the teaching data, so every bucket’s variance is 0 — any real pricing drift would light up immediately.
• Population vs sampling. With 120 values, variance = 218.75 while variance_sampling ≈ 220.59; report the population figure for describing the matched set, the sampling one when estimating beyond it.
• Adjustable strictness. sigma: 3 widens the band for alert-when-truly-extreme; a fractional sigma tightens it for high-sensitivity monitoring dashboards.
• Feed downstream math. The moments are the raw material for z-scores ( normalize z-score) and significance tests ( t_test) without re-scanning the field.

Parameters for extended_stats #

• field
  (Required, string) Numeric field you wish to aggregate.
• sigma
  (Optional, float, default: 2) Multiplier for the standard deviation bounds.
• missing
  (Optional, number) Value used for documents missing the field; it contributes to every accumulator.

Related #

• stats — the five basic numbers without the moments
• percentiles — robust spread (medians) when the mean is skewed
• median_absolute_deviation — a outlier-resistant alternative to σ
• normalize — z-scores built from these moments