Runtime node settings and dynamic log level

Runtime node settings and dynamic log level #

Node-local knobs no longer require a restart. GET/PUT /_node/_local/settings exposes transient overrides over the pizza.yml values — search lane budgets, the slowlog threshold, the default search timeout, trash retention — and they take effect immediately:

# drop the slowlog threshold to 200ms on one node, effective immediately
curl -X PUT http://127.0.0.1:28000/_node/_local/settings \
  -H 'Content-Type: application/json' \
  -d '{"node.search.slow_threshold_ms": 200}'

The flagship: log.level flips live — no restart, no dropped requests — which turns “turn on debug logging on the suspicious node” into a one-liner while preserving target-specific directives like openraft=warn.

Collection-wide manual compaction

Collection-wide manual compaction #

Reclaiming disk no longer means waiting for the background cadence or poking shards one by one. One request runs a merge pass on every started primary copy of the collection, cluster-wide:

curl -X POST http://127.0.0.1:28000/my-collection/_compact \
  -H 'Authorization: Bearer pz_bootstrap_key'

The response reports the queued batch job per hosting node — job id plus how many local primary passes it covers — so a node-level failure fails the row, not the request:

Cluster-wide recycle bin

Cluster-wide recycle bin #

Accidental deletes used to be a race against the sweeper: deleting a namespace only dropped metadata, and the data directories drifted into the trash WITHOUT a recovery snapshot — permanently unrestorable. The recycle bin is now a first-class, cluster-wide surface with a namespace-centric view, a pre-restore integrity check, and one-request restore:

# One cluster-wide list — every node's trash merged by item, grouped by
# namespace, with the delete-time snapshot summary of each item.
GET /_cluster/trash

Deleting a namespace captures a restore snapshot for every collection it contains, so the WHOLE namespace comes back as a unit — original schemas, shard UUIDs, partition routing and placements included:

Host & capability probes — see the machine, not just the node

Host & capability probes — see the machine, not just the node #

Every node now reports what its host actually is and what it can actually do, on GET /_node/_local under host: the OS distribution and kernel, NUMA topology (node count + per-node CPU lists and distances), NIC inventory with negotiated link speeds, the data disk’s bus (NVMe or not) with its I/O scheduler and atime policy, the container and seccomp environment, the DMI platform, transparent-hugepage state — and the resource ceilings: fd limit vs usage, mmap count vs max_map_count, swap, conntrack fill, TCP sysctls, and cgroup CPU/memory quotas with throttle counts. The stats API adds a steal_percent trend — the noisy-neighbor line on shared tenancy.

Lesson 9 — Operations

Lesson 9 — Operations #

Keeping a node healthy: partial updates in place, compaction, the change feed, and reading cluster state. This lesson is server-side — start a node as in Lesson 1 and run the curls against it.

The use case #

The features that matter once the node serves real traffic: mutating documents without rewriting them, keeping storage healthy, reacting to every write, and reading cluster state. The questions this lesson answers:

Trash APIs

Trash APIs #

The trash (recycle bin) is node-local filesystem state — deleted namespaces and collections are moved into <data_path>/trash/ and physically deleted only after trash.retention_secs elapses. These cluster-level APIs aggregate every node’s trash on the server side and give you one namespace-centric view, a read-only shard-completeness verification, and cluster-wide restore/purge:

• List the cluster trash
• Verify a trash item
• Restore a trash item
• Delete a trash item everywhere
• Empty the trash everywhere

Deleting a namespace (DELETE /_namespace/<name>) now moves the whole namespace — every collection, with a restore snapshot per collection — into the trash as a single item; restoring it brings back all collections with their original schemas, shard UUIDs and placements. Deleting a collection keeps the per-collection behavior it always had.