DRBD replication status reference

The SC//Reliant™ platform uses DRBD (Distributed Replicated Block Device) to keep the two nodes of an HA cluster holding identical data. Several cluster procedures ask you to check DRBD status before acting, for example, before performing maintenance on a node. This page explains which command to use on a given node and how to read the output, so you can confirm the nodes are in sync and tell a healthy state from one that needs attention.

Nodes across the fleet do not all offer the same command for this. Read Checking DRBD status before running anything, because the interface available depends on which version of the DRBD utilities the node has.

Checking DRBD status

Which command to use

cat /proc/drbd works on every node the platform currently runs and needs no privileges. Use it when you want one command that behaves the same everywhere, for example in a runbook or an escalation checklist.

cat /proc/drbd

sudo drbdadm status is available on nodes whose DRBD utilities are version 9.1.0 or later, which is the release that added the subcommand for 8.4 kernel modules. Where it is available it is easier to read, because it lists resources by name rather than by minor number.

sudo drbdadm status

To find out which you have, check the utility version. DRBDADM_VERSION is the one that determines whether status exists.

sudo drbdadm --version

The two versions in that output are independent of each other, and both matter:

• DRBDADM_VERSION The version of the drbd-utils package, which supplies the drbdadm command table. On the platform this is either 8.9.x, where status against an 8.4 module returns drbdadm: Unknown command 'status', or 9.x, where it works.
• DRBD_KERNEL_VERSION_CODE The version of the loaded kernel module, as a hexadecimal triplet. For example, 0x08040b is 8.4.11 and 0x080407 is 8.4.7. This determines which states the module can report, and it is also readable from the first line of /proc/drbd.

A node can have a 9.x utility on top of an 8.4 kernel module. On such a node drbdadm status works and prints the 9.x tree format even though the module is 8.4. Do not infer the utility version from the module version, or the reverse.

IMPORTANT: Some upstream DRBD documentation describes /proc/drbd as deprecated in favor of drbdadm status. Do not rewrite platform runbooks on that basis. /proc/drbd is present wherever the kernel module is 8.x, and it is the only interface available on nodes with 8.9.x utilities.

The two output formats

The same underlying information is reported under different field names depending on which command you ran.

The one field that does not map cleanly is the connection state. /proc/drbd reports a single cs: value covering both whether the peer is reachable and whether data is flowing. The tree format splits that into two:

• connection: Whether the nodes are talking. Values include Connected, Connecting, and StandAlone.
• replication: Whether data is moving, and in which direction. Values include Established, SyncSource, SyncTarget, and Off.

So cs:Connected corresponds to connection:Connected together with replication:Established, while cs:StandAlone corresponds to connection:StandAlone. Fields at their default value are omitted from the output, which is why the healthy sample below shows no connection: line.

A healthy resource in the tree format looks like this:

win1 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

The equivalent in /proc/drbd, read from the node holding the secondary copy, is a single line plus a counters line:

50: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:14064 dw:14064 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

The state tables below use the /proc/drbd field names. The state values themselves (Primary, UpToDate, StandAlone, and so on) are the same in both formats.

Roles (ro: local/peer)

A DRBD resource has a role on each node, written local/peer.

Only one node is Primary at a time. ro:Primary/Secondary means the local node is active and the peer is the standby. In the tree format the same state reads as role:Primary with peer role:Secondary.

A node reporting Primary locally and Unknown for its peer, on both nodes at once, is the signature of a split-brain. Each node is claiming the active role while unable to see its peer.

NOTE: All resources reporting Primary on the same node is the desired steady state, not a fault. It means the cluster has its workloads consolidated. Both nodes reporting Primary for the same resource is the problem.

Connection states (cs:)

The connection state describes the replication link to the peer.

Disk states (ds: local/peer)

The disk state describes how current and consistent each copy of the data is, written local/peer.

Common combinations

How a resync works

DRBD does not re-copy the whole disk after a disconnect. While the peer is away, the Primary marks each written extent "dirty" in its out-of-sync bitmap. On reconnect, only those dirty extents are resynced.

DRBD tracks what needs resyncing using three pieces of metadata:

• Generation Identifiers (GI). Markers that let both nodes agree which side has the fresher data after a disconnect.
• Out-of-sync bitmap. Tracks exactly which extents changed while the peer was unreachable, so only those extents resync.
• Activity log. A circular log of recently written extents that enables a fast resync after a clean disconnect.

Reading a live sync

During a resync, the /proc/drbd view for a resource looks like this:

2: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:162850608 nr:0 dw:162843319 dr:1044 al:411567 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:98304
    [=============>......] sync'ed: 78.1% (98304/450560)K
    finish: 0:02:41 speed: 29,752 (29,108) K/sec

The first line reports the resource state:

• 2: The DRBD resource (minor) number.
• cs:SyncSource Connection state. This node is supplying resync data, so the peer is the SyncTarget.
• ro:Primary/Secondary Roles. This node is Primary, the peer is Secondary.
• ds:UpToDate/Inconsistent Disk states. This node is current, the peer is still incomplete.
• C The replication protocol. Protocol C is fully synchronous, meaning a write is acknowledged only after it has reached both nodes' disks.

The remaining lines appear only while a resync is running:

• sync'ed: 78.1% (98304/450560)K Resync progress. The parenthetical is the data still to copy out of the total, here 98,304 KB of 450,560 KB remaining. It matches oos on the counters line, in the same unit. The progress bar is identical on both nodes.
• finish: 0:02:41 speed: 29,752 (29,108) K/sec The estimated time remaining, the current transfer rate, and, in parentheses, the average rate since the resync started.

The counters line reports cumulative activity in KB, or as a count where noted:

Quick interpretation

• Healthy: cs:Connected with ds:UpToDate/UpToDate on both nodes.
• Expected during maintenance: the surviving node shows the peer as WFConnection and/or ds:...DUnknown while the other node is in standby or powered off.
• Wait, do not promote: a node showing ds:Inconsistent or a cs:SyncTarget resync in progress. Let oos reach 0 first.
• Investigate: cs:StandAlone (possible split-brain), an Outdated disk (Pacemaker will not fail over to it), or an oos value that is not decreasing.

Related documentation

• Data volumes and DRBD: the replicated block-device layer.
• Pacemaker and Corosync (high availability): the cluster software that promotes DRBD and coordinates failover.
• Verifying cluster health: the Sensu service checks for overall cluster health.