Verifying a node after maintenance

Because the node's resources were stopped while it was in standby, its DRBD volumes need to catch up with the volumes on the other node before the cluster is fully protected again. This applies to both categories of maintenance, but the scale of the catch-up differs. A short software maintenance window typically produces a small resync that finishes quickly. A hardware window in which the node was powered off for hours produces a much larger one, because DRBD has to ship every extent that changed while the node was away.

Verify the following before you consider the maintenance complete. Run these checks over SSH (see Connecting to a node) on the node you performed the maintenance on, reconnecting first if the work rebooted it. Step 2 is the exception, because DRBD status is reported from the perspective of the node you run it on, so check it on both nodes.

1. Confirm the node has rejoined the cluster and is no longer in standby. Run this on the target node, because Standby and Unclean describe whichever node you are connected to. Both must be false. If you are calling the cluster.status method directly, the fields are standby and unclean.

rvc cluster status

2. Confirm the DRBD volumes have finished resynchronizing. Check this on both nodes, connecting to the other node over SSH if you do not already have a session open there, and use the same command you used to record the baseline. With cat /proc/drbd, both nodes should report cs:Connected with ds:UpToDate/UpToDate, and oos should have fallen to 0. With sudo drbdadm status, look for replication:Established with disk:UpToDate and peer-disk:UpToDate, and add --statistics if you want the out-of-sync figure. For help interpreting either format, see DRBD replication status reference.

cat /proc/drbd

3. Confirm the cluster is healthy. The cluster_maintenance, corosync_resources, corosync_node, and drbd Sensu service checks should all report OK. For more information, including how to retrieve check details through the SC//Reliant API, see Verifying cluster health.

4. If you performed software maintenance, confirm the change actually landed by re-running the version commands from Determining the node's role and recording a baseline and comparing them against what you recorded. Note that rvc sys os reports the running kernel, so after a kernel update it only reflects the new version once the node has rebooted.

rvc sys os

dpkg -l <package_name>

5. If the maintenance rebooted the node, confirm the reboot happened when you expected it to and that the node has been up since.

rvc sys uptime

6. Confirm fencing is back on if you disabled it. In the response, Stonith must read Enabled. The value is Enabled or Disabled, not true or false. Check that both fence devices are started as well, because each node hosts the device belonging to its partner and both stop while a node is in standby.

rvc cluster info

sudo crm_mon -1 -r

7. Close any SSH connections you opened once every check has passed. Do not close them while a resync is still running, because the resync is what returns the cluster to a protected state.

exit

Related documentation

• Performing maintenance on a node: the standby procedure that ends with these checks.
• DRBD replication status reference: how to read DRBD status before and after maintenance.
• Verifying cluster health: the Sensu service checks for overall cluster health.