Node observability APIs #
The /_node/_local/* surface exposes this node’s runtime state —
recovery progress, build backlog, shard inventory, storage layout,
slowlogs, logs and the trash. All of it is read-only except where noted,
and every endpoint can be reached on a remote node through the
unified forward: replace _local with the node id
(/_node/<node_id>/...) — the console’s node pages are
built on this.
Runtime knobs live on the sibling Node Settings page.
Node level #
| Endpoint | What it reports |
|---|---|
GET /_nodes | The cluster’s node inventory. |
GET /_node/_local | This node’s identity, roles and reported addresses. |
GET /_node/_local/stats (aliases /_nodes/stats, /_nodes/_local/stats) | Node-level counters. |
GET /_node/_local/recovery | Shard recovery progress — restart WAL replay and peer pulls, including the queue behind node.recovery_concurrency. The cluster health timeout response points here. |
GET /_node/_local/ready | Readiness probe — answers once startup gates have passed. |
GET /_node/_local/tasks | The unified task view — the single task surface: every maintenance lane (flush, epoch build, recovery, compaction), the batch job registries (flush_jobs[], compaction_jobs[]), the per-shard maintenance tables (build_tasks[], compaction_tasks[]) and engine task-loop health. POST /_node/_local/tasks/flush/<job_id>/cancel cancels a flush job, POST /_node/_local/tasks/compact/<job_id>/cancel a compaction job. |
GET /_node/_local/memory | Memory usage summary. |
GET /_node/_local/memory/profile | A detailed memory profile breakdown. |
GET /_node/_local/logs | Recent node logs — the console’s log viewer. |
Host & capabilities (GET /_node/_local → host)
#
The host section probes the machine behind the node — the facts an
operator would otherwise check over SSH. Everything except the OS
strings, the fd/swap limits and the CPU features is Linux-only and
simply absent on other platforms:
| Field | What it reports |
|---|---|
distribution / os_version / kernel_version | OS identity, e.g. Ubuntu 22.04.4 LTS / 5.15.0-105-generic. |
platform | DMI identity (vendor/product) — which VM or cloud the node runs on. |
container | Container runtime (docker, kubernetes, podman, …); absent = bare metal. |
seccomp | This process’s seccomp mode — filter is the one that usually denies io_uring. |
numa | Node count, per-node CPU lists and distance rows; 1 node means UMA. |
nics / max_link_speed_mbps | NIC inventory with negotiated link speeds (sysfs) and the fastest link — the host’s bandwidth ceiling. Also per-interface speed_mbps on the stats API’s os.net. |
cpu_freq | scaling_driver + scaling_governor — powersave/ondemand on a non-p-state driver is a latency killer. |
cpu_features | Instruction sets available at runtime (avx2, neon, sve, …) — confirms the SIMD paths can engage. |
cpu_vulnerabilities | Per-flaw mitigation state (mitigated / vulnerable / not affected) — PTI and friends tax syscall-heavy paths. |
swap | Swap totals + vm.swappiness. Swap behind mmap’d segments means latency death. |
file_descriptors | RLIMIT_NOFILE (soft/hard) and current usage — the ceiling for sockets + segment mmaps. |
mappings | Current mapping count vs vm.max_map_count (default 65530 — low for many mmap’d segments). |
cgroup | Resource limits (v2 preferred, v1 best-effort): CPU quota in cores, throttle counts (cpu.stat), memory max/high/current, and the sched_getaffinity CPU count. A quota below the engine’s thread count means throttled share-nothing runtimes — size with node.cpus_for_runtime. |
conntrack | conntrack table fill — saturation drops connections silently. |
net_sysctl | TCP ceilings: tcp_rmem/tcp_wmem max, somaxconn, ip_local_port_range. |
transparent_hugepages / transparent_hugepages_defrag / anon_huge_pages_bytes | THP mode, defrag policy, and the THP bytes actually backing the process. |
storage | Data-disk bus (nvme, sata, virtio, …), media verdict, I/O scheduler, and the mount’s atime policy — see also the disk section. |
io_uring | The Fusion-runtime verdict: driver is what the process runs on (io_uring | legacy), status the cause (enabled | forced_legacy | blocked | kernel_missing | ops_missing | platform), detail a human-readable reason. The fallback itself is silent — this is how you see it. See
verify io_uring. |
The stats API (GET /_node/_local/stats) adds one dynamic line to this
picture: os.cpu.steal_percent (and its history.steal_percent[]
trend) — hypervisor-stolen CPU, the noisy-neighbor signal on shared
tenancy.
The console renders all of it: the per-host facts strip (with amber chips for actionable findings) on Infrastructure → Hosts, and the Host / Limits / Network / Runtime-capability panels on each node’s Overview tab.
Shards #
| Endpoint | What it reports |
|---|---|
GET /_node/_local/shards | Every shard copy this node holds, with state, role, and per-shard counters — including reshard_purged (documents purged as superseded by a committed reshard). |
GET /_node/_local/shards/metrics | Per-shard serving metrics (search lanes, queueing). |
GET /_node/_local/shards/<shard_id>/<allocation_id>/files | The shard’s on-disk files. |
GET /_node/_local/shards/<shard_id>/<allocation_id>/storage | The shard’s FIRE storage layout — segments with per-file and per-field costs; feeds the console’s storage explorer. |
GET /_node/_local/shards/<shard_id>/<allocation_id>/epochs | The shard’s epoch ledger (frozen, building, built). |
GET /_node/_local/shards/<shard_id>/<allocation_id>/wal/<epoch_id>[/<seq>] | WAL inspection: an epoch’s entries, or a single entry by sequence number — the crash-forensics view. |
GET /_node/_local/storage | Node-wide storage usage summary. |
Shard maintenance actions #
| Endpoint | Effect |
|---|---|
POST /_node/_local/shards/<shard_id>/<allocation_id>/compact | Run one synchronous compaction pass on this shard copy — the single-shard form; returns when the pass lands. |
POST /_node/_local/shards/<shard_id>/<allocation_id>/compact/cancel | Cancel that copy’s manual compaction: parked passes are drained, a running merge unwinds cooperatively. |
POST /_node/_local/shards/compact_all[?collection=] | Batch form over every local started primary — queues a cancellable job (see /tasks); the node-local form behind the collection-wide
_compact. |
POST /_node/_local/shards/<shard_id>/<allocation_id>/epoch/roll | Freeze the shard’s active epoch now (starts a segment build). |
POST /_node/_local/shards/<shard_id>/<allocation_id>/epoch/flush | Roll the active epoch and build every frozen epoch, then purge the covered WAL. |
POST /_node/_local/shards/flush_all | Batch form of epoch flush across all local shards — runs as a cancellable task (see /tasks). |
POST /_node/_local/shards/<shard_id>/<allocation_id>/doc/<doc_id>/_fields | Apply an inplace fast-lane field update directly on this shard (internal fast-lane form). |
Search observability #
| Endpoint | What it reports |
|---|---|
GET /_node/_local/search_slowlog | Searches slower than node.search.slow_threshold_ms, captured with their full DSL. |
GET /_node/_local/search_audit | The search audit trail (failed/slow/bad requests); DELETE on the same path clears it. |
Trash #
Deleted namespaces and collections (and replaced shards) are moved into
<data>/trash/ and purged after trash.retention_secs (default 1 day)
— see
Backup and restore. These node-local APIs
manage that window for THIS node; the cluster-wide, namespace-centric
view lives under the
Trash APIs
(GET /_cluster/trash aggregates every node server-side):
| Endpoint | Effect |
|---|---|
GET /_node/_local/trash | List trashed items with age, size and snapshot summary (kind/namespace/restorable). |
GET /_node/_local/trash/<item>/verify | Read-only shard-completeness check of this node’s copy (?deep=true also parses every segment file’s footer). |
POST /_node/_local/trash/<item>/restore | Restore a trashed collection — or a whole deleted namespace — cluster-coordinated, gated by the verification above. |
POST /_node/_local/trash/<item>/prepare | Stage a trashed item for inspection. |
GET /_node/_local/trash/<item>/snapshot | Snapshot view of a trashed item’s metadata. |
DELETE /_node/_local/trash/<item> | Physically delete one trashed item now. |
POST /_node/_local/trash/<item>/delete_everywhere | Delete one item on every node’s trash. |
POST /_node/_local/trash/empty_everywhere | Empty the trash on every node. |