Monitoring

Volume health in miroir is reported per node. Each agent exports what its own leg of every volume sees, so a problem shows up as a metric on the node that has it. The controller adds the few cluster-level signals the agents can't know, like RWX gateway health.

monitoring.podMonitor.enabled: true creates a Prometheus Operator PodMonitor scraping the controller and every agent on their metrics ports (the per-volume gauges are exported by the agent on each storage node; a node label is added to every series). The diskful per-volume gauges also carry a pool label naming the pool backing that node's leg. Pools are per-node, so two legs of one volume can report different pools, which lets you scope volume health to a pool (the shipped dashboard's pool variable does exactly that). miroir_volume_diskless_primary and miroir_volume_wedged are the exceptions: a diskless leg holds no backing device in any pool, and a wedged teardown's pool can be unknowable, so both carry no pool label.

Every volume series also carries pvc and pvc_namespace: the PersistentVolumeClaim the volume serves, so dashboards and alerts read the claim's name instead of the opaque pvc-<uuid> volume name (which stays available as the volume label). The pair is recorded on the MiroirVolume at provisioning time and backfilled onto pre-existing volumes from their PV; a volume whose claim is unknown falls back to its volume name in pvc, with an empty pvc_namespace. The agent exports, per volume on that node:

Metric	Meaning
miroir_volume_up_to_date	1 when this node's replica is UpToDate (unreplicated volumes are always 1 once created)
miroir_volume_connected	1 when all replication links to diskful peers are established (tie-breaker links excluded)
miroir_volume_split_brain	1 when DRBD refused to reconnect after divergence; manual resolution required
miroir_volume_suspended	1 while the snapshot write barrier freezes IO; sustained means a stranded barrier
miroir_volume_resync_ratio	fraction (0-1) in sync of the least-synced diskful peer; 1 when fully in sync
miroir_volume_quorum	0 while a freeze volume has lost quorum and refuses writes, the "workloads are failing I/O" signal (always 1 under last-man-standing)
miroir_volume_disk_failed	1 when this leg's disk was detached after an I/O error and latched failed; replace the disk, then remove and re-add the replica
miroir_volume_out_of_sync_bytes	worst per-peer out-of-sync bytes: the exposure if the healthiest peer is lost; also counts online-verify findings
miroir_volume_primary	1 while this node's diskful leg is Primary: the consumer pod or the RWX gateway runs here and this leg serves the I/O
miroir_volume_diskless_primary	1 while a diskless leg (client or tie-breaker) is Primary here: the consumer pays network I/O; see auto-diskful
miroir_volume_verify_last_timestamp_seconds	unix time of the last completed scheduled verify; alert on staleness to catch a schedule that stopped firing
miroir_volume_verify_out_of_sync_bytes	out-of-sync bytes the last scheduled verify found (0 = clean)
miroir_volume_wedged	1 when the kernel can no longer tear down this volume's DRBD resource (LINBIT/drbd#137); only a node reboot clears it

Each agent additionally exports its pool capacities (miroir_pool_capacity_bytes / miroir_pool_allocated_bytes / miroir_pool_meta_used_ratio, one series per named pool via the pool label), the same sample that feeds capacity-aware placement and the PoolUsageHigh condition. Pool exhaustion is alertable, and two pools on one node stay distinguishable, not just an Event. It also exports miroir_node_drbd_kernel_info (always 1): the DRBD kernel module version probed at startup (version label) plus the agent image's drbd-utils version (utils_version label), from client-only nodes too (which have no MiroirNode status). Query it for fleet version skew before a release raises the kernel floor.

Three further node-scoped series cover the failure mode that miroir_volume_wedged only ever sees one volume of at a time:

Metric	Meaning
miroir_node_stranded_children	host commands killed at their deadline whose task is still in uninterruptible sleep, so the kernel will neither run nor reap them
miroir_node_wedged	1 while the node-scoped breaker is open: miroir has stopped spawning storage commands because each new one only strands too
miroir_node_drbd_assertions_total	fatal DRBD kernel assertions (put_ldev refcount underflows, LINBIT/drbd#137) sighted in the kernel log since the agent started

A sustained non-zero miroir_node_stranded_children is the earliest signal that a node's storage stack is jamming: it climbs while the per-volume gauges still look healthy, because the stuck tasks hold kernel locks that commands against healthy volumes then block on. At the breaker's limit miroir_node_wedged goes to 1 and the agent fails further lvm/zfs/drbdsetup calls and unmounts on that node with node storage stack wedged: node reboot required.

Refusing is not a recovery — nothing in userspace can reap a task the kernel holds. It bounds the pile so the node stays drainable: unbounded, every retry (including kubelet's NodeUnstageVolume retries) adds a stuck task until kubelet's own shutdown cannot complete and the node needs an out-of-band power cycle. The shipped MiroirNodeStorageWedged rule alerts on it; drain and reboot that node. The gauge clears by itself if the stuck commands drain. Note that the count lives in the agent process, so restarting the agent resets it — the stuck tasks remain and the breaker re-trips once enough new children strand.

The breaker also opens on a fatal DRBD kernel assertion in the node's kernel log (a put_ldev refcount underflow, LINBIT/drbd#137): the damaged refcount makes the next detach a hang or a use-after-free, so the agent stops initiating DRBD state changes on the node. Unlike the stranded-children trip, this latch never clears on its own and survives agent restarts — the agent replays the kernel ring on start — so miroir_node_wedged holds 1 until the node reboots. Unmounts are not refused on a latch: the filesystem layer still works, and draining the node is exactly the remedy.

For RWX volumes the controller exports miroir_export_ready: 1 while the volume's NFS gateway is serving (gateway pod available, export address published). This is the signal the per-volume gauges cannot give you: DRBD replicas stay healthy while a dead gateway leaves every NFS client hanging.

Each gateway pod additionally serves its own metrics endpoint (scraped by a second PodMonitor, with node and volume labels): miroir_gateway_nfs_healthy is the result of the last liveness probe's NFS NULL call against the pod's local ganesha. The same probe backs the pod's /healthz, so a ganesha that still accepts TCP connections but has stopped answering NFS fails liveness and is restarted. Previously that failure mode was invisible.

Prometheus is where you alert on all of this, and for volume health it is currently the only place. The driver also answers the CSI health RPCs (ControllerGetVolumeHealth, ControllerListVolumeHealth, NodeGetVolumeHealth), reporting a split-brain that needs an operator as DATA_LOSS, a failed disk or lagging replica as DEGRADED, and a volume with no backing device left as INACCESSIBLE. The controller RPCs answer cluster-wide; NodeGetVolumeHealth answers for the node that asks, which is the only place a node-local quorum loss shows up. These replace the VolumeCondition API that CSI v1.13 removed, but on most clusters nothing consumes them yet: kubelet calls NodeGetVolumeHealth only on Kubernetes 1.37+ with the alpha CSIVolumeHealth feature gate enabled (default off), and external-health-monitor has no release that speaks them, so the health-monitor sidecar is gone from the chart (its released versions refuse to start against a driver that no longer advertises VOLUME_CONDITION). Until a consumer runs in your cluster, the miroir_volume_* gauges are what to alert on.

Starter alerts and dashboard

monitoring.prometheusRule.enabled: true ships starter alerts for all of the above (split-brain, quorum lost, stranded barrier, disk failed, a wedged teardown, degraded replication, sustained out-of-sync, an unavailable RWX export, a stale verify schedule, pool and thin-metadata usage, and a down agent). A node whose agent stops answering scrapes loses every miroir_* series, so none of its per-volume alerts can fire; the kernel-floor refusal to start looks exactly like this. monitoring.dashboards.enabled: true installs a Grafana dashboard, either a sidecar-labelled ConfigMap or a grafana-operator GrafanaDashboard CR via monitoring.dashboards.grafanaOperator.

The per-volume alerts inherit the pool label and name the pool in their summaries, so Alertmanager routes and silences can target a single pool. The wedged-teardown alert is the exception: a pool can be unknowable mid-teardown, so its metric carries only volume. The dashboard's pool variable defaults to All; narrowing it filters the volume-health and pool panels together.