Matrix stats aggregation #

A multi-value metrics aggregation computing counts, means, variances, skewness, kurtosis and the correlation matrix across several numeric fields at once.

How it works #

The aggregation works row-wise: a document contributes a row only if it has a value for every listed field (a missing map can fill gaps per field). Documents with any gap are dropped entirely — the reported count is the number of complete rows, and every statistic uses the same rows. At least two rows are required; below that the result is empty.

For each field, over the n complete rows:

• mean — the arithmetic mean;
• variance — the sample variance, dividing by n − 1;
• skewness — the adjusted Fisher–Pearson form n / ((n−1)(n−2)) · Σ((x − mean)/s)³ (0 for a symmetric distribution);
• kurtosis — excess kurtosis Σ((x − mean)/s)⁴ / n − 3 (0 for a normal distribution).

For each pair of fields the result carries the covariance and the Pearson correlation (covariance divided by the product of the two standard deviations; a field with zero variance correlates as 0).

The correlation is linear association only, which is the classic trap with this aggregation: amount in the teaching dataset is exactly price × qty, yet its correlation with price is only ≈0.82 (and with qty ≈0.59). A perfect functional dependence that is not linear does not score 1.

Example #

amount is exactly price × qty on every one of the 120 sales, yet the matrix shows correlation(price, amount) ≈ 0.82, correlation(qty, amount) ≈ 0.59 and correlation(price, qty) ≈ 0.10 — price and qty are nearly independent, and a product is not a linear function of its factors, so no entry reaches 1.

Use cases #

• Spotting derived fields. Two fields correlating at ±1 usually means one is a copy or unit conversion of the other — redundancy worth knowing before indexing or exporting. (And per above, a medium correlation does not clear a field: check for products and ratios.)
• Feature collinearity. Before feeding fields to a model, pull the correlation matrix once: highly correlated inputs add cost and instability without adding information.
• Sanity-checking relationships. If qty and amount historically correlate at ≈0.6 and suddenly move toward 0, either pricing changed or the data pipeline did.
• Shape reconnaissance. Skewness and kurtosis per field flag the heavy-tailed fields where percentiles will say more than a mean.

Parameters for matrix_stats #

• fields
  (Required, string array) Two or more numeric fields to correlate.
• missing
  (Optional, object) Map of field name to the value substituted for documents missing that field — the row is kept instead of dropped.

Related #

• extended_stats — the same moments (variance, skewness, kurtosis) for a single field, without the pairwise matrices.
• stats — count, min, max, mean, sum per field.
• boxplot / percentiles — distribution shape per field when linearity cannot be assumed.