Performing maintenance on a node

The following is a procedure you can follow when you need to perform maintenance on a node. Maintenance falls into two broad categories, and the difference between them determines how much of this procedure applies. Hardware maintenance, such as replacing a failed drive or a power supply, takes the node down. Software maintenance, such as updating the version of Docker or another system package, usually leaves the node powered on and still talking to its peer.

Both categories share the same core procedure. You take the node out of service using standby, which keeps the rest of the cluster running normally, perform the work, and then return the node to service. What changes between them is whether you need to disable fencing first, how long the node is out of service, and what you verify afterward. See Choosing your maintenance approach.

If you need to shut down the whole cluster instead, see Shutting down and restarting a cluster.

NOTE: Standby sets more than one Sensu service check to a failing state. The cluster_maintenance check reports CRITICAL: node $(hostname) is in standby, and because standby also stops the node's DRBD instance, the drbd check reports CRITICAL for the length of the window as well. Checks can report UNKNOWN briefly while the node is converging. All of them return to OK on the first interval after the node is back in service. Because alerts can be routed to downstream ticketing systems, you should perform this procedure inside a planned change window so the alerts are expected. See Verifying cluster health.

Choosing your maintenance approach

Before you begin, decide which category your work falls into. The category tells you which optional steps in this procedure you need.

• Hardware maintenance. Physical work on the node, such as replacing a failed drive, reseating memory, swapping a power supply, recabling, or replacing the chassis. The node is powered off or rebooted, so it stops answering Corosync heartbeats for the duration. The window is often open ended, because it is gated by parts availability and physical access to the site. For the additional steps that a drive or chassis swap needs, see Replacing drives or the chassis.
• Software maintenance. Changes to what is installed or configured on a running node, such as updating the version of Docker or another system package, applying a security patch, or changing a service configuration. The node normally stays powered on and stays a Corosync member throughout. The window is usually short and predictable.

The following table summarizes how the two categories differ in practice.

Whichever category applies, place the node in standby first. Standby is what makes the work safe, because it moves the cluster-managed workloads to the other node before you touch anything. This matters as much for software maintenance as it does for hardware maintenance. A package update that restarts a service Pacemaker is monitoring looks like a resource failure to the cluster, and Pacemaker responds by attempting recovery. If that recovery cannot stop the resource cleanly, it escalates to fencing the node. Evacuating the node first removes the risk entirely, because there are no cluster-managed resources left on it to fail.

Note that standby also stops the node's DRBD resource, because standby makes the node ineligible to run any cluster-managed resource. Replication pauses and the copy on the standby node falls behind the primary for as long as the node is out of service, regardless of which category of maintenance you are performing. That is why every maintenance window ends with a resync. See Verifying a node after maintenance.

When to use maintenance mode instead of standby

Standby depends on a healthy partner node to receive the evacuated workloads. If the other node is already out of service, or if the change has to be applied to both nodes at the same time, there is nowhere for the workloads to migrate to. In that situation, place the cluster in maintenance mode instead. Maintenance mode leaves the workloads running where they are and stops Pacemaker from reacting to the service restarts your change causes.

The tradeoff is that the workloads keep running on the node you are working on, so a service restart affects them directly. Plan for that outage rather than assuming the cluster will absorb it. For the full comparison of the two controls, see Understanding maintenance mode and standby.

IMPORTANT: Maintenance mode does not suppress fencing. It stops Pacemaker from managing resources, but STONITH is still armed and is triggered by membership loss rather than by resource state. If the change reboots the node or restarts Corosync, disable STONITH as well. See Determining whether to disable fencing.

Connecting to the node

Every command in the procedures on this page, apart from ssh itself, runs on a node in the cluster rather than on your own computer, and some of them must run on a particular node. For the prerequisites for shell access, how to connect over SSH, and which node to run each command on, see Connecting to a node.

Determining the node's role and recording a baseline

Prior to doing maintenance on a node in a cluster, you should determine whether the node is considered primary or secondary inside the cluster. The primary node is the one running the cluster's workload resources, such as virtual IP addresses, containers, and virtual machines. Record a baseline at the same time, so you have something to compare against once the work is done.

Both nodes always run some resources, so a node that shows resources in crm_mon is not necessarily the primary. Clone sets run an instance on each node, the DRBD resource runs as Master on one node and Slave on the other, and each node hosts the fence device belonging to its partner, because a node cannot fence itself. What distinguishes the primary is the workload, not the presence of resources.

1. Use SSH to connect to the node you intend to work on. See Connecting to the node.

2. Retrieve cluster details to determine which node in the cluster is considered primary. The primary node is the one where Active Node is true. Do not assume the primary is red0, because either node can hold the resources. If you are calling the cluster.status method directly, the field is active_node. For more information, see Retrieving cluster details.

rvc cluster status

3. Determine the DRBD status of the node. Record the output so you have a baseline to compare against after the maintenance is complete. For help interpreting the output, see DRBD replication status reference.

cat /proc/drbd

NOTE: cat /proc/drbd works on every node. On nodes whose DRBD utilities are version 9.1.0 or later, sudo drbdadm status reports the same information keyed by resource name rather than minor number, which is easier to read. Check with sudo drbdadm --version. Whichever command you use for the baseline, use the same one for the post-maintenance check, because the two commands report the same states under different field names. See DRBD replication status reference.

4. If you are performing software maintenance, record what you are about to change so you can confirm the change afterward and describe the starting point if you need to roll back. Both commands are read only and safe to run at any time.

Record the OS release and the running kernel.

rvc sys os

Record the installed version of the specific package you are changing, for example docker-ce.

dpkg -l <package_name>

Determining whether to disable fencing

If your maintenance powers off or reboots the node, or restarts Corosync or Pacemaker, disable STONITH before you begin so that the surviving node does not fence it, and enable it again once the node is back in service. For how to decide, and the commands to use, see Disabling fencing during maintenance.

Replacing drives or the chassis

Drive and chassis replacement follows the hardware maintenance path, with additional steps for the disk order, the health of the array, and the full DRBD resync that follows. See Replacing drives or the chassis.

Preparing a primary node for maintenance

The main thing to be aware of when performing maintenance on a primary node is to ensure that all of the resources are migrated to the other node in the cluster prior to proceeding. The cluster-managed resources can be migrated by placing the node in standby. For resources outside the cluster, you will need to gracefully shut them down. Use the following steps to prepare a primary node for maintenance.

1. Use SSH to connect to the primary node, which is the node you are working on. Steps 5 and 6 stop workloads on this node, so they have to run here. If you are performing hardware maintenance, open a second terminal and connect to the surviving node as well, so you keep visibility on the cluster when this node goes down. See Connecting to the node.

ssh <node_hostname>

2. Ensure the cluster is not in maintenance mode. The response must be false.

rvc cluster maintenance-mode?

This check matters because maintenance mode suspends Pacemaker's resource management. If you place a node on standby while the cluster is in maintenance mode, the node is marked standby but its resources are stranded on it, which can look like a completed evacuation when it is not. For more information, see Operating the cluster.

3. Place the node in standby. The command produces no output when it succeeds.

sudo pcs node standby <node_hostname>

4. Monitor as all resources are migrated to the other node in the cluster. Wait for the migration to finish before continuing. The -1 flag prints the cluster state once and exits, so run it again to refresh.

sudo crm_mon -1

The node's own status line tells you whether the evacuation has finished. While resources are still moving, the node is listed as standby (with active resources). Once it is empty, the parenthetical disappears and it reads standby. Wait for the plain form before you continue.

Evacuating a primary node is not instant. A cluster-managed virtual machine has to be shut down and started again on the other node rather than moved live, so the whole migration can take several minutes and the virtual machine is unavailable for that time. Everything else waits behind it, which is why the virtual IP and containers can show as Stopped partway through.

5. If you have any containers running outside the cluster, use the following command to stop them. Just replace <container_name> with the name of your container, ensuring that the full name follows the docker-<containername> format.

sudo systemctl stop docker-<container_name>

6. If you have any KVM virtual machines running outside the cluster, use the following command to stop them. Just replace <kvm_name> with the name of your virtual machine.

rvc kvm shutdown --hostname <kvm_name>

7. Perform the maintenance on the node. What this involves depends on the category of work.

• Hardware maintenance. Disable STONITH before you power off or reboot the node. See Determining whether to disable fencing. Your SSH session to the node drops when it goes down, so move to your session on the surviving node. Once the physical work is finished and the node has powered back on, confirm it has rejoined Corosync by running sudo crm_mon -1 on the surviving node, then reconnect to the node over SSH before you continue. If the work involves replacing drives or the chassis, see Replacing drives or the chassis.
• Software maintenance. Confirm with sudo crm_mon -1 that no cluster-managed resources remain on the node, then apply the update. For the general package procedure, see Updating system packages on a node. If the update replaces the kernel, the DRBD packages, LVM, or the cluster software itself, see Updating the cluster software, DRBD, and the kernel before you begin, because those need extra handling. If the update reboots the node, reconnect over SSH afterward.

8. Once you're ready to bring the node back, remove the node from standby.

sudo pcs node unstandby <node_hostname>

9. Start any containers you stopped in step 5.

sudo systemctl start docker-<container_name>

10. Start any KVM virtual machines you stopped in step 6.

rvc kvm start --hostname <kvm_name>

11. If you disabled STONITH before the maintenance, enable it again. Do not skip this step, because the cluster cannot protect itself against a split-brain condition while fencing is off.

sudo crm configure property stonith-enabled=true

12. Verify the node before you consider the maintenance complete. See Verifying a node after maintenance.

NOTE: Removing the node from standby makes it eligible to run resources again, but it does not move the workload back. The node that received the virtual IPs, containers, and virtual machines during the migration remains the primary node until a later event moves them. The node you worked on does pick its other resources back up, including its clone instances, the DRBD Slave, and its partner's fence device, so expect it to show several resources again without being the primary.

IMPORTANT: While the node is in standby, the cluster cannot recover automatically from a failure on either side. The node you are working on is ineligible to run resources and its DRBD copy falls behind, so it cannot take over. The surviving node also loses its fence device for the duration, because that device runs on the node you just evacuated and cannot relocate to the node it fences. A failure during the window therefore means an outage that requires manual recovery. Keep the window as short as the work allows.

Preparing a secondary node for maintenance

Because a secondary node is not running the cluster's workload, preparing it is simpler than a primary node. You only need to place the node in standby so the cluster does not try to fail resources onto it while you work, then perform the maintenance and return it to service. Evacuating it is usually quick, because there is no virtual machine or container to move.

The secondary is not idle, though. It runs its own clone instances, the DRBD Slave, and the primary's fence device, and standby stops all of them. Use the same completion signal as for a primary node: wait for crm_mon to list it as standby rather than standby (with active resources).

If the secondary node runs any containers or KVM virtual machines outside the cluster, those are not managed by Pacemaker and are not migrated by standby. Shut them down gracefully as described in steps 5 and 6 of the primary node procedure, and start them again once the maintenance is complete.

1. Use SSH to connect to the secondary node, which is the node you are working on. If you are performing hardware maintenance, open a second terminal and connect to the primary node as well, so you keep visibility on the cluster when this node goes down. See Connecting to the node.

ssh <node_hostname>

2. Ensure the cluster is not in maintenance mode. The response must be false.

rvc cluster maintenance-mode?

3. Place the node in standby.

sudo pcs node standby <node_hostname>

4. Perform the maintenance on the node. For hardware maintenance, disable STONITH first, and expect your SSH session to drop when the node goes down. See Determining whether to disable fencing. If the work involves replacing drives or the chassis, see Replacing drives or the chassis. For software maintenance, check whether the packages involved need extra handling. See Software maintenance considerations.

NOTE: A secondary node is the natural place to start when a change has to reach both nodes, because it is not running the cluster's resources. If you are updating the cluster software, DRBD, LVM, or the kernel, plan both nodes as a single change before you begin. See Sequencing the update across both nodes.

IMPORTANT: A secondary node is the cluster's failover target. While it is in standby it is ineligible to run resources and its DRBD copy falls behind the primary, so a failure on the primary during your maintenance window means an outage that requires manual recovery rather than an automatic failover. The primary also loses its fence device for the duration, because that device runs on the secondary and cannot relocate to the node it fences. Recovery means removing the secondary node from standby and waiting out a resync before it can be promoted, because DRBD refuses to promote an Outdated disk. Keep the window as short as the work allows, and do not begin maintenance on a secondary node while any DRBD volume is still resynchronizing.

5. Once you're ready to bring the node back, remove the node from standby. If the node rebooted, reconnect to it over SSH first.

sudo pcs node unstandby <node_hostname>

6. If you disabled STONITH before the maintenance, enable it again. Do not skip this step, because the cluster cannot protect itself against a split-brain condition while fencing is off.

sudo crm configure property stonith-enabled=true

7. Verify the node before you consider the maintenance complete. See Verifying a node after maintenance.

Software maintenance considerations

Most system packages can be updated on an evacuated node without any special handling. A few sit underneath the resources the cluster manages, and those need extra care. Because the detail differs a good deal between an ordinary package and the cluster software itself, the two procedures are documented separately.

• Updating system packages on a node. The general procedure for a package on an evacuated node, how a version declared in SC//Reliant Platform Manager differs from one installed directly on the node, updating the Docker Engine, and rolling a package update back.
• Updating the cluster software, DRBD, and the kernel. Corosync, Pacemaker, the DRBD utilities, LVM, and the kernel. Covers why each behaves differently, sequencing the update across both nodes, kernel updates, accounting for a change in the DRBD status command, and what to do when a node does not come back cleanly.

Which page you need depends on the package.

IMPORTANT: Coordinate with Scale Computing Support before updating Corosync, Pacemaker, the DRBD utilities, or LVM on a production node. These components have to interoperate across both nodes of the cluster, so the order the two nodes are updated in matters.

Verifying the node after maintenance

Before you consider the maintenance complete, confirm the node has rejoined the cluster, DRBD has finished resynchronizing, the cluster is healthy, and fencing is back on. For the full list of checks, see Verifying a node after maintenance.

Related documentation

• Updating system packages on a node: the general package procedure, the Docker Engine, and rolling an update back.
• Updating the cluster software, DRBD, and the kernel: Corosync, Pacemaker, the DRBD utilities, LVM, and the kernel.
• Operating the cluster: maintenance mode, standby, and how the two controls interact.
• Connecting to a node: shell access prerequisites, connecting over SSH, and which node to run each command on.
• Disabling fencing during maintenance: when to disable STONITH and how.
• Replacing drives or the chassis: the additional steps for a drive or chassis swap.
• Verifying a node after maintenance: the checks to complete before the maintenance is finished.
• DRBD replication status reference: how to read DRBD status before and after maintenance.
• Verifying cluster health: the Sensu service checks for overall cluster health.
• Shutting down and restarting a cluster: the procedure to use when both nodes need to come down.
• Pacemaker and Corosync (high availability): the cluster software, STONITH, and the failover process.
• Configuring shell access: generating an SSH key and adding it to your account.