Connecting to a node
Most of the commands in the maintenance and update procedures, apart from ssh itself, run on a node in the cluster rather than on your own computer, so you need shell access to the nodes before you start. The commands are a mix of the RVC CLI (rvc), the cluster tools (pcs, crm, crm_mon), and standard system utilities (cat, dpkg, docker), all of which are only available on the node itself.
Prerequisites for shell access
- A terminal. On macOS or Linux, use Terminal. On Windows, use Windows Terminal, PowerShell, or PuTTY.
- Shell access on your user account. By default, a new user does not have shell access. An administrator turns it on for your account.
- An SSH key pair, with the public key added to your account. The platform requires an RSA 4096-bit key. For instructions on generating a key and adding it to your account, see Configuring shell access.
sudoenabled on your shell account. Most of the commands in these procedures requiresudo. Sudo is a separate shell-account setting that defaults to disabled, so having shell access does not by itself grant it. An administrator enables it. For more information, see Understanding shell access settings.- The hostnames of both nodes in the cluster. Node hostnames follow the pattern
sit0001red0, where0001is the site number, so a two-node cluster is a pair such assit0001red0andsit0001red1. You can find node hostnames in the SC//Reliant™ Platform Manager at https://platform.reliant.io/. You need both, because parts of the maintenance procedures are run from the node you are not working on.
Connecting over SSH
Use SSH to connect to the node. Your SSH key authenticates the connection, either from your local agent or from the identity file configured for the host.
ssh <node_hostname>For example, ssh sit0001red0.
WARNING: Do not use agent forwarding (ssh -A) to connect to a node. Agent forwarding does not authenticate you to the node. It gives the node access to the agent running on your own computer, so that the node can authenticate onward to a third host. Anyone with root on that node can use your forwarded agent to impersonate you for as long as the session is open. The nodes are the final hop, so there is nothing to forward the agent for. If you need to reach the second node, open a separate SSH session from your own computer rather than hopping from the first node.
Once connected, the command prompt changes to show the node's hostname. Every command you type from this point runs on the node, not on your own computer. When you are finished, close the connection.
exitWhich node to run each command on
Because a maintenance procedure deliberately takes one node out of service, it matters which node you are connected to when you run a given command. The node you are working on is the target node, and the other node in the cluster is the surviving node.
| Command | Run it from | Why |
|---|---|---|
rvc cluster info, rvc cluster maintenance-mode? | Either node | These report cluster-wide state. |
rvc cluster status | The node whose values you need, which is the target node in the maintenance procedures | Only the Maintenance field is cluster-wide. Hostname, Active Node, Standby, and Unclean all describe the node you are connected to. |
cat /proc/drbd, sudo drbdadm status, sudo drbdadm --version | Each node in turn | DRBD status is reported from the perspective of the node you run it on, and the utility version can differ between the two nodes. Take the baseline on the target node, then use the surviving node to watch the peer state while the target is down and the resync once it reconnects. Note that sudo drbdadm status only exists where the node's DRBD utilities are version 9.1.0 or later. |
sudo pcs node standby, sudo pcs node unstandby | Either node | The command takes the target hostname as an argument, so it does not have to run on the target node. |
sudo crm configure property stonith-enabled=... | Either node | This sets a cluster-wide property. |
sudo crm_mon -1, sudo crm_mon -1 -r | Either node | Any cluster member reports the same cluster state. Use the surviving node whenever the target node is down or about to go down. |
sudo systemctl, dpkg, docker, rvc sys, rvc docker, rvc kvm | The target node | These act on, or report on, the node you are working on. |
NOTE: If your maintenance reboots or powers off the target node, your SSH session to it drops. That is expected. Move to the surviving node to monitor the cluster, and reconnect to the target node once it is back up. For hardware maintenance, connect to the surviving node before you begin so that you do not lose visibility at the moment the target node goes down.
Related documentation
- Configuring shell access: generating an SSH key and adding it to your account.
- Performing maintenance on a node: the standby procedure for primary and secondary nodes.
- Node diagnostic commands: commands for checking a node once you are connected.