Updating system packages on a node

Software maintenance on a node in an SC//Reliant™ platform cluster means changing what is installed on a node that stays powered on and stays a Corosync member throughout. Most system packages need no special handling once the node has been evacuated. A few sit underneath the resources the cluster manages, and those need extra care.

This page covers the general procedure and the packages that are ordinary exceptions to it. For Corosync, Pacemaker, the DRBD utilities, LVM, and the kernel, see Updating the cluster software, DRBD, and the kernel, because those interrupt the machinery the cluster itself depends on.

This page assumes you have already placed the node in standby and recorded a baseline. If you have not, start with Performing maintenance on a node.

Understanding how a package version reaches a node

Packages arrive on a node one of two ways, and which one applies changes how you make the change and how you roll it back.

• Declared in SC//Reliant Platform Manager and applied by Puppet. Some package versions are configuration values rather than something you install by hand. The clearest example is the Docker Engine, whose version is set in the Version field of the Docker or Cluster Docker Resources hash. You change the field, and the next Puppet run installs that version on the node. See Managing SC//Reliant Platform Manager data for containers.
• Installed directly on the node. Anything not represented in Platform Manager is installed with the node's package manager in your SSH session. Additional package sources are themselves configuration, defined in the APT Configurations hash of the node group. See Node group property reference.

Check which case applies before you plan the change, because the Version field is optional and is often left blank. When it holds a value, Puppet enforces it, and a version you install by hand that contradicts it is replaced on the next run. When it is blank, Puppet does not manage the package version at all and a hand-installed version stays. Make the change where the value actually lives.

To see what Puppet intends to do without changing anything, use SSH to connect to the node and run Puppet in no-op mode. Anything it reports as a pending change is what a real run would apply. See puppet.run.

rvc puppet run --noop

IMPORTANT: Puppet runs automatically on every node, so you do not control the timing by default. Scheduled runs happen at the times set by the Cron Puppet Client Hours and Cron Puppet Client Minutes properties, and a catalog run takes a minute or two. That means a value you save in Platform Manager reaches both nodes at the next scheduled run whether you are ready or not. To find when the next run is due, see Configuring automated Puppet runs. See Staging a declared change to one node at a time.

Staging a declared change to one node at a time

Values set at the node group level are shared by both nodes in the cluster, and Puppet applies them on its own schedule. If you save a new package version and do nothing else, the partner node converges at the next scheduled Puppet run. On the node still carrying the workload, that means a package upgrade, and often a service restart, underneath running containers or a virtual machine. Pacemaker sees the resource fail, attempts recovery, and if the resource does not stop cleanly that can escalate to fencing.

Disable Puppet on the partner before you save the change, not after. Saving first leaves a window in which the cron can fire.

1. Use SSH to connect to the node you are not updating and disable Puppet on it. The --message argument is required, because the reason is recorded and shown to anyone who checks. See puppet.disable.

rvc puppet disable --message '<text_describing_reason>'

2. Place the node you are updating in standby, following Performing maintenance on a node.

3. Save the new value in Platform Manager.

4. Apply it to the evacuated node by running Puppet there directly, rather than waiting for cron. See puppet.run.

rvc puppet run

5. Confirm the change landed on the node you updated and not on its partner, by checking the version on both.

6. Return the node to service, then decide what happens to the partner before you re-enable Puppet on it.

IMPORTANT: Re-enabling Puppet on the partner commits it to the declared value at the next run. Only do so once you have decided that the partner should take the change, or after you have reverted the value in Platform Manager. Re-enabling while a value the partner has not taken is still pending recreates exactly the uncontrolled change this procedure exists to avoid.

Use SSH to connect to the partner node to re-enable Puppet on it. See puppet.enable.

rvc puppet enable

NOTE: Do not leave Puppet disabled once the work is finished. The puppet_client Sensu service check warns when Puppet has been disabled for more than a day and goes critical after two. To confirm the state of a node, use rvc puppet disabled?, which returns true when Puppet is disabled. See puppet.disabled?.

Which packages need extra handling

The following table covers the packages that behave differently from an ordinary update. Everything not listed here can be updated on an evacuated node with no additional steps.

Updating a package on an evacuated node

Use this procedure for any package that is not in the extra handling table. The node must already be in standby, and you should have recorded the installed version as part of your baseline.

1. Confirm that no cluster-managed resources remain on the node before you change anything. Do not assume the evacuation finished. The node should be listed as standby, not standby (with active resources).

sudo crm_mon -1

2. Confirm the version you are changing from, if you did not already record it. Substitute the package you are updating for <package_name>.

dpkg -l <package_name>

3. Apply the update. For a version declared in Platform Manager, edit the value and run Puppet against the evacuated node. For a package installed directly on the node, install it in your SSH session. Pinning the version explicitly, rather than taking whatever is current in the repository, keeps the result predictable and gives you an exact version to return to.

sudo apt install <package_name>=<version>

4. Confirm the new version is installed.

dpkg -l <package_name>

5. If the update restarted a service, confirm the service came back. Substitute the unit you expect to have restarted for <service_name>.

sudo systemctl status <service_name>

6. Return the node to service and verify it. See Verifying a node after maintenance.

NOTE: A package update does not usually need STONITH disabled, because the node keeps answering Corosync heartbeats throughout. Disable it if the update installs a new kernel, requires a reboot, or restarts Corosync or Pacemaker. See Determining whether to disable fencing.

Updating the Docker Engine

Restarting the Docker daemon stops every container on the node, both the containers Pacemaker manages and the ones running outside the cluster. On a node you have already evacuated, the cluster-managed containers are running on the other node, so the restart only affects the containers you stopped when you prepared the node.

The Docker Engine version can be a declared value rather than something you install by hand. It is set in the Version field of the Docker or Cluster Docker Resources hash, described in Platform Manager as the Docker package version. The field is optional, so check whether it holds a value before deciding how to make the change. For the field itself, see Managing SC//Reliant Platform Manager data for containers.

1. Confirm the evacuation finished, so you know the restart cannot affect a cluster-managed container. The node should read standby, not standby (with active resources).

sudo crm_mon -1

2. Apply the change and let the node converge. If you are setting the version in Platform Manager, stage it so the partner does not converge at the same time. See Staging a declared change to one node at a time.

3. Confirm the new daemon version on the target node.

docker version

The version reported here is not identical to the package version. A package such as 20.10.15~reliant is reported by docker version as 20.10.15, without the distribution suffix. Compare the package form against dpkg -l docker-ce and the plain form against docker version, rather than expecting the two to match exactly.

4. Confirm that the containers you stopped when you prepared the node are running again. The cluster-managed containers do not appear here yet, because the node is still in standby and they are running on the other node.

rvc docker containers

NOTE: rvc docker stop is not a substitute for evacuating the node. If the cluster is not in maintenance mode, a container stopped that way is restarted by Pacemaker. See docker.stop.

Two container-level details are worth separating from the Docker Engine itself. Changing the image tag a container runs is a container change rather than a node package update, and it does not require the node to be evacuated. See Managing containers. Reclaiming disk space taken by superseded images after a round of updates is a scheduled task rather than part of this procedure. See Using container image pruning.

Rolling back a package update

Plan the rollback before you apply the update, while the node is still evacuated and you still have the window. A node in standby is the cheapest place to reverse a change, because the workloads are already running elsewhere.

• For a version declared in Platform Manager, set the field back to the version you recorded in your baseline and run Puppet against the node again. Do not reinstall the old version by hand and leave the declared value pointing at the new one, because the next Puppet run reverses your rollback.
• For a package installed directly on the node, reinstall the recorded version, pinning it explicitly.

sudo apt install <package_name>=<version>

• Confirm the rollback landed the same way you confirmed the update, then verify the node as you would after any maintenance. See Verifying a node after maintenance.

IMPORTANT: Reverting a package is not always enough to undo its side effects. An update that migrated an on-disk format, rewrote a configuration file, or changed a data directory layout can leave changes behind that reinstalling the older version does not reverse. Check the vendor's release notes for the version you are moving to before you assume the rollback is clean, and contact Scale Computing Support if a package has changed state that the older version cannot read.

Before you start a fleet-wide update

The procedure on this page updates one node. Rolling the same change across a fleet raises questions this page does not answer, in particular how updates are staged, what the supported version window is, and when a change is large enough to stop being a package update at all. Confirm the rollout plan with Scale Computing Support before you begin a fleet-wide change.

Moving a node to a new SC//Reliant platform generation is a different operation called an in-place upgrade, and Scale Computing performs it on your behalf rather than through this procedure. If you are not sure which side of that line a particular change falls on, contact Scale Computing Support or your account manager before you begin.

Related documentation

• Performing maintenance on a node: the parent procedure, including standby, fencing, and verification.
• Updating the cluster software, DRBD, and the kernel: the packages that need more than an evacuated node.
• Managing SC//Reliant Platform Manager data for containers: the Version field and the container hashes.
• Node group property reference: the APT Configurations hash for additional package sources.
• Using container image pruning: reclaiming disk space taken by superseded images.
• Operating the cluster: maintenance mode, standby, and how the two controls interact.
• puppet.disable: disabling Puppet on a node, and the required reason.
• puppet.enable: returning a node to automated configuration management.
• puppet.run: running Puppet on demand, including no-op mode.