Updating the cluster software, DRBD, and the kernel

Updating the cluster software, DRBD, and the kernel

Corosync, Pacemaker, the DRBD utilities, LVM, and the kernel need more care than an ordinary package update on an SC//Reliant™ platform node. Every other package sits on top of the cluster. These sit underneath it, so updating them interrupts the machinery the cluster itself depends on to decide whether a node is healthy.

For packages that are not in this category, see Updating system packages on a node. This page assumes you have read Performing maintenance on a node and understand standby, fencing, and the verification steps.

IMPORTANT: Coordinate with Scale Computing Support before updating Corosync, Pacemaker, the DRBD utilities, or LVM on a production node. These components have to interoperate across both nodes of the cluster, so the order the two nodes are updated in matters, and a version that works on one node is not guaranteed to talk to the version on the other. Support can confirm the target versions and the order for your cluster before you open a change window.

Why these packages are different

Each of these components fails in its own way when it is interrupted, and the differences determine which precautions you need.

ComponentWhat happens when it restarts or reloadsConsequence
CorosyncThe node drops out of cluster membership.The surviving node sees the node as failed and can fence it. Disable STONITH first.
PacemakerThe node leaves resource management but keeps its Corosync membership.Resources on the node are no longer managed while it is down. Membership is intact, so fencing is less likely, but a service that does not stop cleanly can still be read as a failure.
DRBD utilitiesThe command line tools change, and the status interface the node offers can change with them.Your pre-maintenance baseline and your post-maintenance check can end up in two different output formats.
DRBD kernel moduleReloaded on reboot, and bound to the running kernel.If no module is available for the new kernel, DRBD does not load and the node cannot rejoin replication.
LVMThe volume group and logical volume tooling underneath the DRBD devices changes.The devices DRBD replicates are logical volumes, so this sits one layer below the cluster's storage.
KernelRequires a reboot, which drops Corosync membership and reloads the DRBD module.Treat the work as hardware maintenance for the purposes of fencing and DRBD verification.

Updating the pcs package is not in this category. It restarts pcsd, which serves the pcs command line and its configuration API but holds no Corosync membership and manages no resources, so the restart does not disturb the cluster. Treat it as an ordinary package. See Updating system packages on a node.

Before you begin

Work through the following before you open the change window. Each item prevents a failure mode that is expensive to recover from once the node is down.

  • Confirm the target versions with Scale Computing Support. Both nodes have to interoperate during the window, when one is updated and the other is not.
  • Confirm no DRBD volume is resynchronizing. A resync in progress means the cluster is not currently protected, and taking a second node out of service compounds it. See DRBD replication status reference.
  • Record the DRBD utilities version on the target node. You need it to interpret your own baseline afterward.
sudo drbdadm --version
  • Record the OS release and running kernel.
rvc sys os
  • For a kernel update, confirm a DRBD kernel module is available for the target kernel before you reboot. If no matching module is present, DRBD does not load after the reboot and the node cannot rejoin replication. This is the single check most worth doing in advance, because discovering it after the reboot leaves you with an unprotected cluster and a node that cannot be promoted.
  • Plan for both nodes. These updates eventually have to reach both nodes to leave the cluster on matching versions. See Sequencing the update across both nodes.

Sequencing the update across both nodes

A cluster whose two nodes disagree about these versions is a temporary state you pass through, not one to leave in place. Plan both nodes as a single change, run them one after the other, and let the cluster return to a fully protected state between them.

Update the secondary node first and the primary node last. The secondary node is not running the cluster's workload, so the window on it is cheaper and evacuating it is quicker, and if a version turns out to be incompatible you find out on the node that is not serving anything. Updating the primary node last also means the cluster fails over at most once.

The secondary is not idle, though. It runs its own clone instances, the DRBD Slave, and the primary's fence device, and standby stops all of them. Replication pauses for the length of the window on either node, so neither update is free.

  1. Update the secondary node, following Preparing a secondary node for maintenance.

  2. Return it to service and wait for DRBD to finish resynchronizing before you touch the other node. Do not begin the second node while the first is still catching up, because the cluster has no protected copy until the resync completes.

cat /proc/drbd
  1. Confirm the cluster is healthy and that both nodes agree on the new version.
rvc cluster info
  1. Update the primary node, following Preparing a primary node for maintenance. Placing it in standby moves the resources to the node you already updated.

NOTE: Removing a node from standby does not move the workload back. After updating both nodes, the node that was originally secondary is running the virtual IPs, containers, and virtual machines, and the primary role has transferred to it permanently until a later event moves it again. That is expected, and there is no need to move anything back. Each node does pick its own clone instances, DRBD role, and partner fence device back up. See the note at the end of Preparing a primary node for maintenance.

When the change has to hit both nodes at once

Standby depends on a healthy partner node to receive the evacuated workloads, so it cannot be used when the change has to be applied to both nodes at the same time. In that situation, place the cluster in maintenance mode instead. The workloads keep running where they are, and Pacemaker stops reacting to the service restarts your change causes.

Plan for the outage rather than assuming the cluster absorbs it, because the workloads are running on the nodes you are working on. And remember that maintenance mode does not suppress fencing. If the change restarts Corosync or reboots either node, disable STONITH as well. For the full comparison of the two controls, see Understanding maintenance mode and standby.

Updating the kernel

A kernel update requires a reboot, so treat it as hardware maintenance rather than software maintenance for the purposes of fencing and DRBD verification. The node stops answering Corosync heartbeats, and the DRBD kernel module is reloaded on the way back up.

  1. Confirm a DRBD kernel module is available for the target kernel. Do not skip this. Without a matching module, DRBD does not load after the reboot.

  2. Disable STONITH, because the reboot guarantees the node drops out of cluster membership. See Determining whether to disable fencing.

sudo crm configure property stonith-enabled=false

  1. Apply the update and reboot the node. Your SSH session drops when it goes down, which is expected. Move to your session on the surviving node to watch the cluster.

  2. Reconnect to the node once it is back up, and confirm it has rejoined Corosync.

sudo crm_mon -1
  1. Confirm the DRBD module loaded and the node has rejoined replication. A node that comes back without DRBD shows the resource down rather than resynchronizing.
cat /proc/drbd
  1. Confirm the new kernel is running. Note that rvc sys os reports the running kernel, so it only reflects the new version after the reboot.
rvc sys os
  1. Enable STONITH again once the node is back in service. Do not skip this, because the cluster cannot protect itself against a split-brain condition while fencing is off.
sudo crm configure property stonith-enabled=true
  1. Verify the node fully before moving on to the other one. See Verifying a node after maintenance.

Accounting for a change in the DRBD status command

Updating the DRBD utilities can change which status command the node offers, which means your recorded baseline and your post-maintenance check may be in two different output formats.

cat /proc/drbd works on every node and keys its output by minor number. sudo drbdadm status reports the same information keyed by resource name, which is easier to read, and exists only where the node's DRBD utilities are version 9.1.0 or later. A node moving from 8.9.x to 9.1.0 or later gains it. Older material uses sudo drbd-overview, which was long deprecated and was removed in drbd-utils 9.6.0, so do not rely on it being present.

Re-check the version on the target node after the update.

sudo drbdadm --version

If the format changed, use DRBD replication status reference to map between the two. The two commands report the same states under different field names, so the comparison against your baseline is still valid once you have translated it.

NOTE: If you know in advance that the update crosses the 9.1.0 boundary, take your baseline with cat /proc/drbd. It is available on both sides of the change, which saves you translating between formats at the point where you most want a direct comparison.

Verifying the update

Verify the node as you would after any maintenance, using Verifying a node after maintenance. The following additional checks are specific to these packages.

  1. Confirm the cluster software is running and the node holds membership. Both nodes should appear online.
sudo crm_mon -1
  1. Confirm the DRBD volumes reached a fully synchronized state rather than stopping partway. Check this on both nodes.
cat /proc/drbd
  1. Confirm fencing is back on. In the response, Stonith must read Enabled. The value is Enabled or Disabled, not true or false. Confirm both fence devices are started too, since each node hosts its partner's device and both stop while a node is in standby.
rvc cluster info
sudo crm_mon -1 -r

  1. Confirm the corosync_resources, corosync_node, drbd, and cluster_maintenance Sensu service checks all report OK. See Verifying cluster health.

  2. Confirm both nodes agree on the new versions once the second node is done. A mismatch left in place is the condition this whole procedure is sequenced to avoid.

dpkg -l <package_name>

If the node does not come back cleanly

  • The node rejoins Corosync but DRBD does not load. The most likely cause after a kernel update is that no DRBD kernel module was available for the new kernel. The node holds membership but cannot replicate, so the cluster is unprotected. Do not force a promotion. Contact Scale Computing Support.
  • The DRBD volumes come up Outdated and do not resynchronize. DRBD refuses to promote an Outdated disk, so the node cannot take over resources in this state. See DRBD replication status reference for reading the state, and treat it as a recovery rather than as part of the maintenance window.
  • Both nodes report themselves primary, or the volumes will not reconnect. This is a split-brain condition, and it needs a deliberate recovery rather than a retry of the update. Do not run the update again on either node. Contact Scale Computing Support.
  • The node is fenced during the window. Confirm whether STONITH was left enabled for a change that dropped Corosync membership. Re-check the guidance in Determining whether to disable fencing before you retry.

© 2026 Scale Computing, Inc. All rights reserved.

Scale Computing, SC//AcuVigil, SC//Connect, and SC//Reliant are trademarks of Scale Computing, Inc.