Changes

Release notes

/ Changes

Release notes #

Information about release notes of INFINI Pizza is provided here.

Latest #

v0.1 is pending — nothing has been released yet. Everything below is the full feature set heading into the first release, in active development.

v0.1 (pending) #

Hello World, Hello Pizza！The first release: a distributed, real-time, AI-native search engine.

Distributed core #

  • Cluster catalog metadata management, synchronized through Raft (openraft, with a monoio runtime binding)
  • RPC over Cap’n Proto, zero-copy both ways
  • Multi-region organization for large clusters
  • Promotion hardening — WAL catch-up barrier, primary-less sweep, idempotent role swaps; membership self-fence closes the zombie-primary window
  • Online rolling reshard — progress and events API, fast-drain orchestration, hardlink staging
  • WAN-safe Raft timing knobs; join tokens and API keys persisted through the catalog Raft; network.advertise split from binding
  • Formal verification — the consistency-critical protocols (promotion, allocation, reshard, doc-id reservation, layered visibility, fencing) are model-checked in TLA+; counterexamples have caught 18+ real bugs

Realtime engine and storage #

  • True real-time indexing — no refresh, no flush; documents are searchable the moment they are acked
  • WAL durability, local and Kafka-backed
  • FireV11 segment format with a compaction pipeline, maintenance states, and task-lane metrics; per-collection manual compaction API with batched jobs
  • Partial field updates (_update) — toggle operator, optimistic concurrency over RPC, replicated fast lane, inplace checkpoints, and inplace typed columns that turn hot counter / flag updates into O(1) column writes
  • Trash / recycle-bin lifecycle with a sweeper
  • Memory profile dump endpoint; deletion-ledger rebuild at mount (documents can no longer resurrect after restart)

Query engine #

  • BM25 scoring with execution explain
  • Lucene query syntax and the Elasticsearch QueryDSL surface
  • Distributed query execution with cross-shard aggregation reduce
  • Point-in-time (PIT) pinned snapshot reads, ES pit API
  • Supported query types: match, match_all, match_phrase, multi_match, term, terms, bool, range, prefix, fuzzy, wildcard, regexp, suffix, exists, span, query_string, collapse, nested, join and traverse (relations), a graph projection block that renders any hit set as nodes and edges, geo_bounding_box, geo_distance, geo_polygon, geo_shape, vector/multi_vector (kNN), hybrid, semantic, and streaming search
  • Supported field types: keyword, text, booleans, all fixed-width signed/unsigned integers (i8…u64), floats and doubles, dates, arrays, objects, dense and sparse vectors
  • Rhai scripting — script_fields on the search request, script_score and script filters in queries, pipeline-aggregation scripts and scripted metrics

SQL and PRQL #

  • POST /_sql — a SQL-92-shaped SELECT subset over collections, lowered into the same engine pipeline as the DSL: projection with expressions, aliases and CASE, filters with three-valued NULL logic (IN/BETWEEN/LIKE/IS [NOT] NULL), GROUP BY/HAVING with expression keys and DATE_TRUNC/HISTOGRAM bucketing, all five aggregates with DISTINCT, ORDER BY/LIMIT/OFFSET, and grouped keyset cursor pages (see the syntax walkthrough)
  • $1 positional parameters bound on the parsed tree (one cached plan serves every param set), PIT snapshot pinning, per-request timeout, and a 256 KiB statement cap; plan errors answer with 1-based source positions
  • Pizza extension functions inside SQL — MATCH, MATCH_PHRASE, FUZZY, KNN, QUERY (raw DSL predicate), SCORE() and HIGHLIGHT
  • POST /_prql — PRQL pipelines compiled by a pinned prqlc onto the same /_sql pipeline; the concise leading-pipe form is accepted, and compile errors carry positions mapped onto the original PRQL text
  • The SQL frontend lives in the engine itself (feature sql) — the gateway wire and the WebAssembly build ride the same parse/lower, so the two hosts cannot drift
  • Server-side embedding inference — text→vector at index and query time through any OpenAI-compatible embedding service
  • AI services registry — embedding model entries with provider presets, credentials resolved through the node keystore
  • semantic query — semantic search with zero field knowledge: no vector field, no dimensions, no model config in the query body
  • TurboQuant 4-bit quantized vector search, with compaction and rolling-reshard support end to end
  • sparse_vector fields over HTTP, indexing through search

Elasticsearch compatibility #

  • ES retriever/knn hybrid syntax translated into the native DSL
  • Execute or reject, never skip — a strict stage gate and top-level key whitelist: unsupported ES syntax fails loudly instead of silently degrading
  • Keyed writes replace in place, render ES-style updated/200; ?key_as_id=true surfaces the user key as _id; delete-by-key included

Text analysis #

  • Built-in standard and whitespace analyzers, configurable stopwords, CJK-aware tokenization, lowercase/uppercase normalizers
  • Analysis workbench in the console — catalog, pipeline builder, stage-by-stage debugging with per-stage diffs
  • 40+ language analyzer family (contrib/analysis-*): ICU segmentation, jieba/SmartCN/IK/pinyin/stconvert for Chinese, lindera (kuromoji/nori) for Japanese and Korean, the European stemmer set, and more

Security #

  • Two-plane role model — built-in roles, API token lifecycle with one-time reveal, namespace-scoped key minting
  • Authn/authz middleware across the data plane and write path

Observability and operations #

  • Search audit ring with _node endpoints; request-id correlation middleware
  • Per-node ES-shaped stats API and golden-metric trends
  • Node task manager — batch flush, priority gate, unified lane view
  • Node-local keystore for secrets — ${keystore:} config references, CLI management, masked view
  • Runtime node settings with dynamic log level; process RSS watchdog; recovery-time API and concurrency controls

Web console #

  • Redesigned home with live cluster telemetry, topology and monitoring views, a FIRE storage explorer with per-field cost treemap, WAL entry drill-down, tasks tabs with compaction controls, a settings editor, and an AI services panel
  • Web CLI — curl → console DSL conversion on paste, multi-select Run-N batch execution with stacked results
  • SQL/PRQL runs directly in the console drawer (⌘/Ctrl+Shift+O): bare statement blocks with keyword and schema completion, typed table results with a raw-JSON toggle, $1 params and PIT snapshot pinning behind the sql options popover, cursor paging via the table’s Next page button, and statements recorded distinctly in history

Tools and playground #

  • Pizza CLI — quick interaction with the Pizza server
  • In-browser playground — the same engine compiled to WebAssembly, mounting real .fire segments over a deterministic teaching dataset; every tutorial query runs live ( try it)
  • Docs site with dark mode, tutorials, a full reference sweep, and headless smoke tests asserting documented examples against the dataset

Change feed (CDC)

/ Changes, / Cdc, / Replication

Change feed (CDC) #

GET /_changes serves a collection’s change log tail from the shard engines’ WALs, so external systems can replicate Pizza data elsewhere — off-site disaster recovery, secondary indexing, audit pipelines — with a standard poll loop:

GET /demo:orders/_changes?since=abc:3,def:1&limit=1000

{
  "results": [
    {"shard":"abc…","rolling_id":0,"epoch_id":4,"seq":0,
     "id":"0,7","key":"u1","op":"create","doc":{"qty":3}},
    {"shard":"abc…","rolling_id":0,"epoch_id":4,"seq":1,
     "id":"0,8","op":"delete"}
  ],
  "cursors":     {"abc…": 4, "def…": 1},
  "purged_below": {"abc…": 3},
  "active":      {"abc…": 5},
  "pending":     true
}

Request #

GET /_changes
GET /<targets>/_changes

Path parameters #

  • targets
    (Optional, string) Comma-separated collection names (wildcard supported); the target-less form feeds every collection.

Query parameters #

  • since
    (Optional, string) Per-shard epoch cursors from a previous response, shard_hex:epoch,shard_hex:epoch. Omit to start from the beginning of the retained tail.
  • limit
    (Optional, integer) Maximum rows to return. Defaults to 1000.

Consuming the feed #

  1. Poll with the cursors of the previous response as the next since (persist the cursors — they are the checkpoint).
  2. pending: true means more rows exist beyond limit; poll again immediately, otherwise poll at your cadence.
  3. Compare each cursor against purged_below: a cursor below that floor means the WAL retention window (wal.local.retention_epoch / retention_bytes) has already purged part of the tail for that shard. The tail is incomplete — run a full resync (search + re-bulk into the DR cluster) and resume from the current cursors.

graph TD P("poll — since = the cursors you saved") --> R["rows + new cursors"] R -->|"pending: true — rows remain past limit"| P R --> C{"every cursor at or above
purged_below?"} C -->|"yes — the tail is complete"| W["persist the cursors,
poll again at your cadence"] C -->|"no — retention already purged
part of the tail"| X["full resync: search + re-bulk,
resume from the current cursors"]

...

©INFINI.LTD, All Rights Reserved.