Replacing drives or the chassis

Replacing drives or the chassis

Drive and chassis replacement follows the hardware maintenance path, with three additional considerations: the order the disks go back in, confirming the array is healthy before the node rejoins the cluster, and the resync that a node with replaced disks has to complete before the cluster is protected again. Read this page alongside the procedure for the node's role rather than instead of it. See Preparing a primary node for maintenance and Preparing a secondary node for maintenance.

Moving disks into a replacement chassis

If you are swapping the chassis and carrying the existing disks over, put them back in the same slot order they came out of. A hardware RAID controller identifies array members by slot as well as by the metadata written to the disk, so an array whose members appear in an unexpected order can be reported as foreign or degraded even though every disk is present and healthy. Label the disks as you remove them rather than relying on memory.

Verifying the array before the node rejoins the cluster

Check the local storage from the system BIOS, the RAID BIOS, or the hardware RAID manager before the node boots far enough to rejoin the cluster. Confirm the following.

  • Every expected drive is present.
  • The virtual disk or RAID set reports healthy or optimal, or is rebuilding as expected.
  • No array is reported as failed, foreign, or missing members.

If any of those checks fail, stop and resolve the storage problem before you let the node continue. Cluster recovery on top of an unhealthy array produces a second failure on the node you were repairing, and DRBD starts replicating onto storage that is about to fail again.

A rebuilding array is not a reason to stop, but it is a reason to wait. Let the RAID rebuild finish before you remove the node from standby, so that the node is not serving a DRBD resync from an array that is still reconstructing itself.

Returning a node with replaced disks to the cluster

A node whose disks were replaced comes back with no DRBD metadata on them. That is a different situation from a split-brain condition, where both nodes hold a copy and the two have diverged. Here there is no replication history on the returning node at all, and the surviving node is unambiguously the authoritative copy.

The node's configuration run provisions DRBD on the replacement disks once it is back up, so the metadata does not need to be recreated by hand. What follows is a full resync rather than the incremental one an ordinary maintenance window produces, because DRBD has no out-of-sync bitmap to work from and has to copy every extent from the surviving node. Expect it to take proportionally longer, and treat the cluster as unprotected until it finishes.

While the resync runs, the returning node reports itself as the sync target holding an inconsistent copy, and reports the surviving node as up to date. Progress is shown as a percentage for each resource.

watch -n 5 'cat /proc/drbd'

The resync is complete when every resource reports cs:Connected with ds:UpToDate/UpToDate and oos has fallen to 0. For help reading either output format, see DRBD replication status reference.

IMPORTANT: If the replacement disks do not pick up their DRBD configuration, or a resource stays disconnected after the node has rejoined Corosync, contact Scale Computing Support rather than recreating the metadata yourself. Wiping and recreating DRBD metadata is a Level 3 procedure, it is not the standard fix for a replaced node, and running it against the node that holds the good data can destroy the only remaining copy.

© 2026 Scale Computing, Inc. All rights reserved.

Scale Computing, SC//AcuVigil, SC//Connect, and SC//Reliant are trademarks of Scale Computing, Inc.