Serial diff aggregation #

A parent pipeline aggregation computing the difference between each bucket’s metric and the bucket lag positions back — a differencing transform that turns a trending or seasonal series into its own period-over-period change.

"wowl": { "serial_diff": { "buckets_path": "revenue", "lag": 2 } }

How it works #

serial_diff is declared inside the multi-bucket aggregation it decorates, next to the metric it consumes. After the buckets are built, the engine walks them in order and attaches current − value_lag_buckets_back to each bucket under the pipeline’s own name:

"buckets": [
  { "key_as_string": "2024-01-01", "doc_count": 1,
    "revenue": { "value": 15 }, "wowl": { "value": null } },
  { "key_as_string": "2024-01-05", "doc_count": 1,
    "revenue": { "value": 30 }, "wowl": { "value": null } },
  { "key_as_string": "2024-01-09", "doc_count": 1,
    "revenue": { "value": 75 }, "wowl": { "value": 60.0 } }
]

Details verified against the engine:

• The first lag buckets have nothing lag positions behind them and answer null.
• lag defaults to 1 and is clamped to a minimum of 1. With lag: 1 the result is the bucket-to-bucket change — exactly what derivative computes.
• buckets_path addresses a metric in the same bucket: a bare name ("revenue"), optionally with a .value suffix; "_count" differences document counts.
• A bucket whose metric is missing — or whose counterpart lag buckets back is missing — answers null; other buckets are unaffected.
• Bucket order is the parent’s output order — chronological for a date_histogram, which is what makes the lag meaningful.
• A second, engine-specific placement also works: declaring serial_diff next to the bucket aggregation, with buckets_path: "per_day>revenue", returns a standalone buckets array holding only the buckets from index lag onward — the same numbers, outside the parent’s response.

Executed example: over the series 60, 150, 150 with lag: 2 the answers are null, null, 90.

Examples #

Each wowl is how the day’s amount moved against the selling day two buckets earlier. Switch the interval to "month" with lag: 1 for month-over-month deltas: the teaching dataset’s 16 monthly revenues (270, 495, 405, …) difference to +225 for February 2024, −90 for March, down to the year’s biggest drop, −270, in December.

Use cases #

• Period-over-period change — monthly revenue with lag: 1 is the month-over-month delta every status meeting asks for, computed with the series, not after it.
• De-seasonalize — set lag to the season length (7 on daily buckets, 12 on monthly ones) so a repeating cycle cancels out and only the underlying trend remains.
• Growth alarms — difference the series and alert on the deltas: a sudden negative spike names the exact bucket where the series broke.
• Volume changes — buckets_path: "_count" differences document counts, isolating traffic shifts from per-document value shifts.

Parameters for serial_diff #

• buckets_path
  (Required, string) Path to the consumed metric in the same bucket.
• lag
  (Optional, integer, default: 1) How many buckets back to compare; clamped to a minimum of 1.
• gap_policy
  (Optional, string) skip (default) or insert_zeros.
• format
  (Optional, string) Output number format. Accepted for compatibility; not applied to the value in the current engine.

Related #

• derivative — the lag: 1 special case
• cumulative_sum — the inverse direction, accumulating instead of differencing
• moving_avg, moving_fn — smooth first when the raw series is noisy