Skip to content
miroir

Troubleshooting

  • Agent pod CrashLoopBackOff on lvmthin: partition or disk missing, or dm_thin_pool not loaded. Check kubectl logs -n miroir-system -l app.kubernetes.io/component=agent and lsmod | grep dm_thin on the node. On a multi-pool node the agent only exits when every pool fails setup; a single bad pool is logged and quarantined (its volumes error, the other pools keep serving) and shows up in the MiroirNode status as a per-pool message.
  • Agent pod CrashLoopBackOff on loopfile: baseDir isn’t reflink-capable. The agent refuses to start (single-pool node) so the failure shows up immediately.
  • Agent pod CrashLoopBackOff after a node change: the DRBD kernel module may be below the agent’s floor (see Requirements); the agent refuses to start rather than render options the module rejects. The agent log names the probed version and the floor.
  • PVC stays Pending: every node with a MiroirNode is missing or full. kubectl describe pvc shows the controller’s reason.
  • Replicated volume stuck in Degraded: one leg isn’t UpToDate. kubectl describe miroirvolume <name> shows per-node status; usually a transient DRBD sync. If it outlasts the sync, with the Inconsistent leg parked WFBitMapT (or Established) toward an UpToDate peer and zero out-of-sync on both sides, DRBD 9.3.4 ended the resync on the source without the target noticing (LINBIT/drbd#148; seen after a rolling node upgrade with a diskless Primary writing in #576). The Inconsistent leg’s agent cycles the connection within a couple of poll cycles and emits StuckResyncRecovered, forcing the disconnect if a read-only opener pins the device (the kernel refuses a graceful one with Need access to UpToDate data); the re-run handshake starts the resync. The agent holds off while a Primary exists that this leg cannot see: that is an ordinary unstable resync, which DRBD re-runs itself once the Primary is back in view.
  • Replicated volume stuck Connecting, no split-brain: a host-network tenant (commonly the Ceph mgr dashboard) occupies the DRBD replication port; dmesg shows Failed to initiate connection, err=-98. Set drbd.portBase (e.g. 7100) to move miroir’s range; existing volumes keep their ports. Full forensics in #148. The same err=-98 with no fixed tenant on the port means the range overlaps the kernel’s ephemeral ports (net.ipv4.ip_local_port_range widened to start below drbd.portBase): any outbound host-network socket can then hold a replication port, and the peer parks StandAlone until the next drbdadm adjust (issue #501). Either keep the ephemeral range above miroir’s ports or reserve them with net.ipv4.ip_local_reserved_ports.
  • MiroirVolumeOutOfSync firing while everything reads healthy: out-of-sync bits toward a peer with no resync draining them. With the connection Connected and both disks UpToDate, this is one of two things. A stale bitmap — bits stranded by a refused clear during peer teardown, left on a newly promoted Primary toward a Secondary after a failover, left between two Secondaries after a connectivity blip while the Primary stays clean toward both, or a resync DRBD armed and abandoned after a rapid promote/demote — is detected and self-healed as long as the volume has a live Primary: the agent cycles the affected peer connection within a couple of poll cycles and emits a StuckResyncRecovered event; the re-run handshake discards the bitmap (identical data moves nothing) or starts the resync it called for. A drbdadm verify finding (lastVerifyOutOfSyncBytes non-zero in the coordinator’s status slot, VerifyOutOfSync event) is a genuine data difference and is deliberately left manual — auto-resyncing would destroy the evidence of which leg was wrong. The same hold applies while a verify is in flight (verifyStartedAt set on the coordinator’s slot until its result is recorded), since the kernel may already hold findings no status field accounts for. Inspect first, then find the affected peer with drbdsetup status <res> --verbose --statistics on the alerting node (the connection whose out-of-sync is non-zero) and cycle it: drbdsetup disconnect <res> <peer-node-id> followed by drbdsetup connect <res> <peer-node-id> resyncs the flagged blocks from the UpToDate side.
  • Resync activity on every kopiur backup: kopiur’s staged PVC inherits the source PVC’s StorageClass, so a replicated volume gets a replicated (and therefore syncing) staging volume once per backup cycle. Point spec.staging.storageClassName at a replicas: "1" class naming the same pool, per Stage kopiur backups unreplicated.
  • RWX MiroirVolume stuck deleting, Device is held open: a still-running gateway is the live opener. The controller scales the share Deployment to zero itself when the volume is deleted, so check kubectl get deploy -n miroir-system miroir-share-<volume> first. If it is still at 1 replica (controller not running, or a release from before it did this), scale it to zero or delete it so the agent’s teardown finalizer can finish. If it is already at 0 but the gateway pod stays Terminating, the pod is wedged on the device itself and the Deployment is no longer the lever: inspect the DRBD resource on that node as for a stuck RWO volume. See ReadWriteMany (RWX).