Derivative aggregation #

A parent pipeline aggregation computing the difference between each bucket’s metric and the previous bucket’s — “how much did it change”. A time series says where you are; the derivative says which way and how fast you are moving, bucket by bucket. The first bucket has no predecessor and answers null.

"delta": { "derivative": { "buckets_path": "revenue" } }

How it works #

derivative is declared inside the multi-bucket aggregation it decorates (the parent style), next to the metric it consumes. After the buckets are built, the engine walks them in order and attaches current − previous to each bucket under the pipeline’s own name:

"buckets": [
  { "key_as_string": "2024-01-01", "doc_count": 1,
    "revenue": { "value": 15 }, "delta": { "value": null } },
  { "key_as_string": "2024-01-05", "doc_count": 1,
    "revenue": { "value": 30 }, "delta": { "value": 15.0 } }
]

Details verified against the engine:

• buckets_path addresses a metric in the same bucket: a bare name ("revenue"), optionally with a .value suffix ("revenue.value"). "_count" differences the bucket document counts.
• A bucket whose metric is missing gets a null derivative, and the next bucket is differenced against the last non-null value — a single gap does not break the chain.
• Bucket order is the parent’s output order — for a date_histogram that is chronological, which is what makes the differences meaningful.
• The unit parameter (report the derivative per day/week in a date_histogram) is not implemented: no per-time-unit scaling, and no normalized_value in the response. A derivative over daily buckets is “per bucket”, which is per day only if every bucket is one day wide.
• gap_policy only takes effect in the sibling placement below; the nested form always answers null for gaps as described above.

A second, engine-specific placement also works: declaring derivative next to the bucket aggregation, with buckets_path: "per_day>revenue", returns a standalone buckets array (keys 1..n−1) holding the differences — the same numbers, outside the parent’s response.

Examples #

Every day in the dataset sells exactly once, so each delta is simply how the day’s amount moved against the previous selling day.

Use cases #

• Day-over-day change — daily revenue with its derivative: growth days read positive, dips negative, zeros flat.
• Spike detection — a delta far beyond its neighbors flags the bucket where a series jumped (an outage’s recovery, a campaign’s lift) without fetching the raw series.
• Growth per period — calendar_interval: "month" buckets and their derivatives give month-over-month deltas of any per-month metric.
• Volume shifts — buckets_path: "_count" differences document counts: when did traffic itself, not the average, move?

Parameters for derivative #

• buckets_path
  (Required, string) Path to the consumed metric in the SAME bucket ("revenue").
• gap_policy
  (Optional, string) skip (default) or insert_zeros — effective in the sibling placement; the nested form answers null on gaps.
• format
  (Optional, string) Output number format. Accepted for compatibility; not applied to the value in the current engine.
• unit
  Not implemented — accepted for compatibility and ignored.

Related #

• cumulative_sum — the running total each bucket feeds into; serial_diff — the difference lag buckets back, for detrending
• moving_fn — window statistics over the same series