How realtime works

Mon, 01 Jan 0001 00:00:00 +0000

How realtime works? #

Do you like to let your customer wait? #

[Pizza in Realtime]

A document written to Pizza is searchable the moment its write is acknowledged — there is no refresh, no near-real-time window, and nothing to tune to get there. This page walks the write path that makes that true.

The write path #

WAL first. The write is appended to the shard’s write-ahead log (a local file-based WAL by default; a Kafka-backed WAL is available). Durability follows wal.local.durability — strict fsyncs per append, batched (the default) group-commits by size and timeout, async skips the hot-path fsync entirely.
Mutable layer. The live documents are indexed into the shard’s in-memory mutable layer, which is itself searchable — this is what removes the refresh: reads see the mutable layer and the immutable segments in one snapshot, so the visibility point is the write’s acknowledgement.
Epoch freeze and build. When the active epoch crosses a size or ops threshold (wal.local.epoch_freeze_max_bytes / epoch_freeze_max_ops, runtime-tunable), it is frozen and built in the background into an immutable FIRE segment. The build does not gate reads or writes.
WAL purge. Once a segment is durably built (per wal.local.segment_durability), the WAL entries it supersedes are purged, bounded by the retention window (wal.local.retention_epoch / retention_bytes).

Recovery is the mirror image: on restart, a shard replays its WAL tail over its last durable segments, so acknowledged writes survive crashes. Restart replay and peer recovery are both bounded by node.recovery_concurrency.

Streaming search

Mon, 01 Jan 0001 00:00:00 +0000

Streaming #

Search on Pizza is real-time — documents are visible the moment they are written. The stream block attaches delivery directives to a search request: how the engine should pace and batch the emission of a result that outgrows one response, instead of making the client poll.

POST /my-collection/_search

How it works #

stream is a native, parsed block of the Pizza request DSL (StreamConfig in lib/engine/src/search/request/mod.rs) — accepted, validated, and carried through the engine with every request. Its knobs describe the emission strategy of the search pipeline: