Percentiles aggregation #

A multi-value metrics aggregation that calculates one or more percentiles over numeric values extracted from the aggregated documents.

Percentiles show the point at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values.

Percentiles are often used to find outliers. In normal distributions, the 0.13th and 99.87th percentiles represents three standard deviations from the mean. Any data which falls outside three standard deviations is often considered an anomaly.

When a range of percentiles are retrieved, they can be used to estimate the data distribution and determine if the data is skewed, bimodal, etc.

How it works #

Two estimators are available; the hdr option selects which one runs.

TDigest arithmetic (default). The engine collects every value of the field, sorts it, and treats each value as a weight-1 centroid sitting at the quantile (i + 0.5) / n — the atomic-digest case of the TDigest. A requested percentile is read off by piecewise-linear interpolation between the two adjacent centroids that bracket it, clamped to the minimum and maximum at the extremes. The row-based and columnar execution paths implement the identical function. When a query fans out, the raw sorted values travel to the coordinator and the quantiles are recomputed there with the same math — parallelism costs no accuracy.

HDR histogram (hdr). Setting hdr: {} switches the estimator: every value is first quantized onto an HDR (High Dynamic Range) recording grid — with the default 3 significant digits, 1024 sub-buckets per binary exponent, anchored at the smallest recorded value. A percentile is then the round(p/100 · n)-th smallest recorded value, clamped to [1, n]: a rank lookup with no interpolation, because the grid has already done the smoothing. HDR cannot represent non-positive values; such values are dropped before quantization. The trade-off between the two estimators is fidelity versus predictability: TDigest interpolates between observed values, while HDR snaps to a fixed relative-resolution grid that stays stable as values stream in.

By default the result is a map keyed by percentile ("50.0"); the keyed flag below switches it to an array of {key, value} objects.

Examples #

Where do the sale prices cluster? The default [1, 5, 25, 50, 75, 95, 99] ladder, live over the teaching dataset’s 120 sales:

Assume your data consists of website load times. The average and median load times are not overly useful to an administrator. The max may be interesting, but it can be easily skewed by a single slow response.

Let’s look at a range of percentiles representing load time:

POST latency/_search
{
  "aggs": {
    "load_time_outlier": {
      "percentiles": {
        "field": "load_time"
      }
    }
  }
}

By default, the percentile metric will generate a range of percentiles: [1, 5, 25, 50, 75, 95, 99]. The response will look like this:

{
  ...
 "aggregations": {
    "load_time_outlier": {
      "values": {
        "1.0": 10.0,
        "5.0": 30.0,
        "25.0": 170.0,
        "50.0": 445.0,
        "75.0": 720.0,
        "95.0": 940.0,
        "99.0": 980.0
      }
    }
  }
}

As you can see, the aggregation will return a calculated value for each percentile in the default range. If we assume response times are in milliseconds, it is immediately obvious that the webpage normally loads in 10-725ms, but occasionally spikes to 945-985ms.

Often, administrators are only interested in outliers — the extreme percentiles. We can specify just the percents we are interested in (requested percentiles must be a value between 0-100 inclusive):

POST latency/_search
{
  "aggs": {
    "load_time_outlier": {
      "percentiles": {
        "field": "load_time",
        "percents": [95, 99, 99.9]
      }
    }
  }
}

Use cases #

• Tail latency. For service latency, the mean is dragged around by a handful of slow calls. The p95 and p99 of a load_time-style field are the standard way to state “almost every request is at least this fast”, and the default ladder gives you both without extra plumbing.
• Outlier detection. Values beyond the 0.13th and 99.87th percentiles (three standard deviations, in normal distributions) are conventional anomaly candidates — cheaper than keeping every raw value around.
• Skew-aware reporting. Over the teaching dataset’s 120 sale prices the mean is 27.5 but the median ("50.0") is 22.5 — half of all sales cost the minimum 15. When mean and median disagree like this, quote the median.
• Distribution reconnaissance. The default ladder [1, 5, 25, 50, 75, 95, 99] is a five-second sketch of shape: equal spacing suggests symmetry, gaps suggest modes. Follow up with boxplot or histogram to zoom in.
• Fixed-resolution latency histograms. HDR (hdr: {}) trades a little accuracy for a value grid that does not move as more data arrives — useful when successive time buckets must be compared side by side.

Parameters for percentiles #

• field
  (Required, string) Field you wish to aggregate.
• percents
  (Optional, array) A range of percentiles that are calculated. Default is [1, 5, 25, 50, 75, 95, 99].
• missing
  (Optional, number) Value substituted for documents without the field.
• hdr
  (Optional, object) Switches to the HDR histogram (High Dynamic Range quantization) estimator instead of the default TDigest, with its resolution configured by number_of_significant_value_digits.

keyed #

By default the keyed flag is set to true which associates a unique string key with each percentile and returns the values as a hash rather than an array. Setting the keyed flag to false will disable this behavior:

POST latency/_search
{
  "aggs": {
    "load_time_outlier": {
      "percentiles": {
        "field": "load_time",
        "keyed": false
      }
    }
  }
}

Response:

{
  ...
  "aggregations": {
    "load_time_outlier": {
      "values": [
        {
          "key": 1.0,
          "value": 10.0
        },
        {
          "key": 5.0,
          "value": 30.0
        },
        {
          "key": 25.0,
          "value": 170.0
        },
        {
          "key": 50.0,
          "value": 445.0
        },
        {
          "key": 75.0,
          "value": 720.0
        },
        {
          "key": 95.0,
          "value": 940.0
        },
        {
          "key": 99.0,
          "value": 980.0
        }
      ]
    }
  }
}

Related #

• percentile_ranks — the inverse question: for a given value, what percentage of documents sit at or below it.
• boxplot — the quartiles (25/50/75) plus whiskers, in one result.
• median_absolute_deviation — a robust spread measure built on the median.
• extended_stats — moments and standard-deviation bounds, the parametric counterpart to these order statistics.