Deployment

Deployment #

Pizza ships as a single statically-linked binary — copy it, give it a data directory, run it. This page covers the deployment shapes; for the full configuration key reference see configuration and pizza.yml reference.

Single node #

The default pizza.yml runs a self-contained node: metadata catalog, shards and the HTTP API in one process.

./pizza -c pizza.yml

Data lives under path.data (default ./data). The binary needs no external services — no ZooKeeper, no external metastore.

Multiple instances on one machine #

Each instance needs a distinct API port, data directory and advertise address. The configuration reference documents the detection rules; in short, pass them explicitly:

./pizza -c pizza.yml \
  -E api.port=28001 \
  -E path.data=./data-2 \
  -E cluster.region.advertise_addr=127.0.0.1:9301

Multi-node cluster #

Run one node, then join the rest with a join token minted on the first node:

curl -X POST -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"note": "bootstrap worker"}' \
  http://seed:28000/_cluster/join_token

Start additional nodes with the returned token and the seed address; the cluster management reference has the full workflow. Replication and resharding then require no further action — see sharding.

Systemd unit #

[Unit]
Description=Pizza search node
After=network-online.target

[Service]
ExecStart=/opt/pizza/pizza -c /etc/pizza/pizza.yml
Restart=on-failure
LimitNOFILE=65535
# NUMA: pin the node to one socket on multi-socket hosts
# numactl --cpunodebind=0 --membind=0 /opt/pizza/pizza -c /etc/pizza/pizza.yml

[Install]
WantedBy=multi-user.target

Pizza is share-nothing per CPU — prefer one process per NUMA socket over one big process on multi-socket machines (see architecture).

Verify io_uring is really active #

Pizza’s engine runs on a monoio Fusion runtime: io_uring where the kernel allows it, an automatic (and silent) fallback to the legacy driver where it doesn’t. The fallback does not change functionality — it gives up the io_uring performance path. On Linux it is worth verifying once after deployment that the fast path is actually in use:

strace -f -e trace=io_uring_setup,io_uring_enter -p <pid>

Recurring io_uring_setup / io_uring_enter calls while traffic flows mean it is live; complete silence means the node is on the fallback.

Common causes of the fallback, in rough order of frequency:

  • Container seccomp profiles — older Docker default profiles and some Kubernetes-restricted profiles deny io_uring_setup. Docker 20.10+ with its default profile allows it; custom profiles may need io_uring_setup, io_uring_enter, io_uring_register added via securityContext.seccompProfile or an updated profile.
  • gVisor (runsc) and some cloud sandboxes — user-space kernels that do not implement the io_uring syscalls at all.
  • Kernels older than 5.10 — io_uring exists from 5.1 but the engine requires the opcode coverage of 5.10+; older kernels get the legacy driver.
  • MONOIO_FORCE_LEGACY_DRIVER=1 in the environment (explicit opt-out).

You do not need strace to notice the fallback: each node reports its verdict on GET /_node/_local under host.io_uring — driver is what the process runs on (io_uring | legacy), status classifies the cause (enabled | blocked | kernel_missing | ops_missing | forced_legacy), and detail is a human-readable reason. The console shows the same on Infrastructure → Hosts (per-host io_uring / legacy IO badge) and on each node’s Overview tab, which also lists the NUMA topology, NIC link speeds, NVMe-or-not for the data disk, and the container/seccomp environment — the host-level checks that used to require an SSH session.

Host checks #

The same host section surfaces the classic search-engine performance traps. The console’s Hosts page shows an amber chip for each one that applies; the quick fix list:

FindingFix
swap.total_bytes > 0Disable swap (or vm.swappiness=1) — swapped mmap’d segments are latency death.
atime is relatime/strictatime on the data mountMount the data volume noatime — every atime update is a hidden write tax on segment reads.
cpu_freq.governor is powersave/ondemand on a non-p-state driverSet the performance governor (intel_pstate/amd-pstate active mode are fine as-is).
fd usage (file_descriptors) above ~80%Raise LimitNOFILE (systemd) or the shell ulimit — sockets and segment mmaps share this ceiling.
mmap count (mappings) near vm.max_map_count (default 65530)Raise vm.max_map_count (e.g. 1048576) — segment mounts fail when it caps out.
conntrack (conntrack) above ~80%Raise net.netfilter.nf_conntrack_max — a full table drops connections silently.
cgroup CPU quota below the node’s engine thread countSize node.cpus_for_runtime to the quota/container CPU limit — thread-per-core runtimes inside a small quota throttle each other.

Capacity tuning baseline #

Measured on a 2-node cluster at 64-way client concurrency (2026-10-01 tuning matrix): engine width and the search admission cap dominate behavior under load.

  • node.cpus_for_runtime: keep the default (empty = every core). Pinning a single engine core caps both writes and searches — 1 → 4 cores was a 3.6× throughput difference. List specific cores only to fence the engine away from front-end cores.
  • node.search.max_per_shard: scale with engine width (rule of thumb: 16 × engine cores). An undersized cap degrades over-cap shards as network failures during load spikes.
  • region.fault_detection.follower_check.threshold: raise (e.g. 30000 ms) on slow or heavily loaded test hardware to ride out recovery storms without false failed-node detections.
  • The all-green stress baseline: every core for the engine + the scaled per-shard cap + the raised follower-check threshold.

Verify io_uring is active (Linux) #

The internal plane runs on monoio, which prefers io_uring and silently falls back to the legacy driver where the kernel or the sandbox forbids it (some container seccomp profiles do). Check a running node once after deployment:

strace -f -e trace=io_uring_setup,io_uring_enter -p <pizza-pid>

Ring setup/enter calls streaming by = io_uring active; nothing = the legacy driver is serving (functionally identical, no io_uring benefit). Durable-write fsync offload has its own switch: PIZZA_WAL_IO_URING=0 disables it (check the startup log for “io_uring fsync offload active”). | cpu_throttled_ms climbing (cgroup) | Same class: the container is CPU-starved for its configured concurrency. | | steal_percent elevated (stats trend) | Noisy neighbor on shared tenancy — move the VM or reserve dedicated cores. |

Upgrades and rollback #

  • Replace the binary and restart; on-disk formats are forward-compatible within a minor series (see release notes).
  • The startup gate keeps the node answering 503 (with phase detail) until recovery completes — put / or /_node/_local/ready behind your load balancer health check.
  • For a full discussion of backup and restore (Raft snapshots, trash) see backup.
Calendar October 1, 2026
Edit Edit this page