Deployment #
Pizza ships as a single statically-linked binary — copy it, give it a data
directory, run it. This page covers the deployment shapes; for the full
configuration key reference see
configuration
and
pizza.yml reference.
Single node #
The default pizza.yml runs a self-contained node: metadata catalog, shards
and the HTTP API in one process.
./pizza -c pizza.yml
Data lives under path.data (default ./data). The binary needs no external
services — no ZooKeeper, no external metastore.
Multiple instances on one machine #
Each instance needs a distinct API port, data directory and advertise address. The configuration reference documents the detection rules; in short, pass them explicitly:
./pizza -c pizza.yml \
-E api.port=28001 \
-E path.data=./data-2 \
-E cluster.region.advertise_addr=127.0.0.1:9301
Multi-node cluster #
Run one node, then join the rest with a join token minted on the first node:
curl -X POST -H "Authorization: Bearer $KEY" \
-H 'Content-Type: application/json' \
-d '{"note": "bootstrap worker"}' \
http://seed:28000/_cluster/join_token
Start additional nodes with the returned token and the seed address; the cluster management reference has the full workflow. Replication and resharding then require no further action — see sharding.
Systemd unit #
[Unit]
Description=Pizza search node
After=network-online.target
[Service]
ExecStart=/opt/pizza/pizza -c /etc/pizza/pizza.yml
Restart=on-failure
LimitNOFILE=65535
# NUMA: pin the node to one socket on multi-socket hosts
# numactl --cpunodebind=0 --membind=0 /opt/pizza/pizza -c /etc/pizza/pizza.yml
[Install]
WantedBy=multi-user.target
Pizza is share-nothing per CPU — prefer one process per NUMA socket over one big process on multi-socket machines (see architecture).
Verify io_uring is really active #
Pizza’s engine runs on a monoio Fusion runtime: io_uring where the kernel allows it, an automatic (and silent) fallback to the legacy driver where it doesn’t. The fallback does not change functionality — it gives up the io_uring performance path. On Linux it is worth verifying once after deployment that the fast path is actually in use:
strace -f -e trace=io_uring_setup,io_uring_enter -p <pid>
Recurring io_uring_setup / io_uring_enter calls while traffic flows
mean it is live; complete silence means the node is on the fallback.
Common causes of the fallback, in rough order of frequency:
- Container seccomp profiles — older Docker default profiles and
some Kubernetes-restricted profiles deny
io_uring_setup. Docker 20.10+ with its default profile allows it; custom profiles may needio_uring_setup,io_uring_enter,io_uring_registeradded viasecurityContext.seccompProfileor an updated profile. - gVisor (
runsc) and some cloud sandboxes — user-space kernels that do not implement the io_uring syscalls at all. - Kernels older than 5.10 — io_uring exists from 5.1 but the engine requires the opcode coverage of 5.10+; older kernels get the legacy driver.
MONOIO_FORCE_LEGACY_DRIVER=1in the environment (explicit opt-out).
You do not need strace to notice the fallback: each node reports its
verdict on GET /_node/_local under host.io_uring — driver is what
the process runs on (io_uring | legacy), status classifies the
cause (enabled | blocked | kernel_missing | ops_missing |
forced_legacy), and detail is a human-readable reason. The console
shows the same on Infrastructure → Hosts (per-host io_uring /
legacy IO badge) and on each node’s Overview tab, which also lists
the NUMA topology, NIC link speeds, NVMe-or-not for the data disk, and
the container/seccomp environment — the host-level checks that used to
require an SSH session.
Host checks #
The same host section surfaces the classic search-engine performance
traps. The console’s Hosts page shows an amber chip for each one that
applies; the quick fix list:
| Finding | Fix |
|---|---|
swap.total_bytes > 0 | Disable swap (or vm.swappiness=1) — swapped mmap’d segments are latency death. |
atime is relatime/strictatime on the data mount | Mount the data volume noatime — every atime update is a hidden write tax on segment reads. |
cpu_freq.governor is powersave/ondemand on a non-p-state driver | Set the performance governor (intel_pstate/amd-pstate active mode are fine as-is). |
fd usage (file_descriptors) above ~80% | Raise LimitNOFILE (systemd) or the shell ulimit — sockets and segment mmaps share this ceiling. |
mmap count (mappings) near vm.max_map_count (default 65530) | Raise vm.max_map_count (e.g. 1048576) — segment mounts fail when it caps out. |
conntrack (conntrack) above ~80% | Raise net.netfilter.nf_conntrack_max — a full table drops connections silently. |
| cgroup CPU quota below the node’s engine thread count | Size node.cpus_for_runtime to the quota/container CPU limit — thread-per-core runtimes inside a small quota throttle each other. |
Capacity tuning baseline #
Measured on a 2-node cluster at 64-way client concurrency (2026-10-01 tuning matrix): engine width and the search admission cap dominate behavior under load.
node.cpus_for_runtime: keep the default (empty = every core). Pinning a single engine core caps both writes and searches — 1 → 4 cores was a 3.6× throughput difference. List specific cores only to fence the engine away from front-end cores.node.search.max_per_shard: scale with engine width (rule of thumb: 16 × engine cores). An undersized cap degrades over-cap shards as network failures during load spikes.region.fault_detection.follower_check.threshold: raise (e.g. 30000 ms) on slow or heavily loaded test hardware to ride out recovery storms without false failed-node detections.- The all-green stress baseline: every core for the engine + the scaled per-shard cap + the raised follower-check threshold.
Verify io_uring is active (Linux) #
The internal plane runs on monoio, which prefers io_uring and silently falls back to the legacy driver where the kernel or the sandbox forbids it (some container seccomp profiles do). Check a running node once after deployment:
strace -f -e trace=io_uring_setup,io_uring_enter -p <pizza-pid>
Ring setup/enter calls streaming by = io_uring active; nothing = the
legacy driver is serving (functionally identical, no io_uring benefit).
Durable-write fsync offload has its own switch: PIZZA_WAL_IO_URING=0
disables it (check the startup log for “io_uring fsync offload active”).
| cpu_throttled_ms climbing (cgroup) | Same class: the container is CPU-starved for its configured concurrency. |
| steal_percent elevated (stats trend) | Noisy neighbor on shared tenancy — move the VM or reserve dedicated cores. |
Upgrades and rollback #
- Replace the binary and restart; on-disk formats are forward-compatible within a minor series (see release notes).
- The startup gate keeps the node answering 503 (with phase detail) until
recovery completes — put
/or/_node/_local/readybehind your load balancer health check. - For a full discussion of backup and restore (Raft snapshots, trash) see backup.