Deploying changes to your environment

Deploying changes to your environment

Configuration changes you make in the SC//Reliantâ„¢ Platform Manager are not applied to your nodes at the moment you save them. The Platform Manager stores the desired configuration, and Puppet is what deploys it. The properties set at the site, node group, and node level are compiled into a Puppet catalog, and a Puppet agent on each node applies that catalog on the next Puppet run. Until Puppet runs on a node, that node continues running its previous configuration.

A change reaches a node in one of two ways. Either you wait for the next scheduled Puppet agent run, or you trigger a Puppet run yourself. This page covers triggering a run using the SC//Reliant API and the SC//Reliant Virtual Controller (RVC) CLI.

Automated Puppet runs are driven by a cron schedule that you control through node group properties, so the schedule varies from one environment to another. Do not assume a particular interval when planning a deployment. To find out when the next scheduled run is due on a node running RVC 3.13.0 or later, use rvc puppet status and read the Next Run At value. On earlier RVC versions, use rvc sys crontab to see the schedule. For more information about the settings that govern the schedule, see Configuring automated Puppet runs.

IMPORTANT: The procedures on this page show you how to run Puppet. They do not define a deployment strategy. Deciding which nodes to deploy to, in what order, and in what change window is specific to your environment and the change you are making. Deploying a configuration change to every node in a large estate at once is rarely what you want. Determine your target set and sequencing before you begin.

Before you deploy

Confirm the following before triggering a run.

  1. Your change is saved in the SC//Reliant Platform Manager. For node group changes, this means you completed the impact review and confirmation. For more information, see Updating a node group.
  2. You know which nodes the change affects. When you update a node group, the impact screen reports the number of affected nodes.
  3. Puppet is enabled on the target nodes. A node with Puppet disabled does not apply the catalog. To check, use rvc puppet disabled?, which returns true when Puppet is disabled.
  4. No Puppet run is already in progress on the target nodes. With RVC 3.13.0 or later, use rvc puppet status and confirm State is stopped. On earlier versions, rvc puppet run refuses to start while another run is in progress and exits with code 1.

Previewing a change before you deploy it

Puppet can simulate a run without applying anything. This is called noop mode, and it reports what the run would change while leaving the node untouched. Use it to confirm a change does what you expect before you deploy it for real, particularly on the first node of a larger rollout.

Using the RVC CLI:

rvc puppet run --noop

Using the SC//Reliant API:

ruby api.rb <customer_code> post api/jobs/puppet '{"action":"runonce","noop":true,"hostnames":["<node_hostname>"]}'

A noop run is a safe way to verify targeting and catalog contents. It is not a guarantee, because a real run can still fail on a node whose live state differs from what the simulation assumed.

Deploying using the SC//Reliant API

The API creates a Puppet job that runs against the targets you specify. This is the method to use when deploying to more than one node, because a single request can target a node group or a saved filter rather than a list of hostnames. Puppet runs are started through the Jobs v1 Puppet endpoint, api/jobs/puppet, with the runonce action. The Jobs v2 RVC endpoint does not run Puppet.

In order to perform this action, the following conditions must be met:

  • Must have the Remote Job Execution feature turned on for your SC//Reliant Platform Manager tenant.
  • The user making the API request must have the Create and Update a Puppet Run job capability.

Deploying to specific nodes

Target one or more nodes by hostname.

ruby api.rb <customer_code> post api/jobs/puppet '{"action":"runonce","hostnames":["<node_hostname>"]}'

Deploying to a node group

Target a node group to run Puppet on all of its member nodes. This is the natural counterpart to updating a node group, because the target set matches the set of nodes the change affects.

ruby api.rb <customer_code> post api/jobs/puppet '{"action":"runonce","node_group_name":"<node_group_name>"}'

Deploying to a saved filter

Target a saved filter to run Puppet on all nodes that match it. Use this when your target set does not correspond to a single node group.

ruby api.rb <customer_code> post api/jobs/puppet '{"action":"runonce","saved_filter_name":"<saved_filter_name>"}'

Retrieving the result

The API response confirms the job was created and scheduled, not that Puppet finished. Use the job ID from the response to retrieve the result.

ruby api.rb <customer_code> get api/jobs/<job_id>\?include_payload=true

The payload confirms that a Puppet run was started on each node. It does not report the outcome of the run. To check the outcome, see Verifying a deployment, or start a job with the last_run_summary action. For more information, see Retrieve a Puppet last run summary.

For the full request and response reference, including the environment parameter, see Run the Puppet agent once.

Deploying using the RVC CLI

Running Puppet from the RVC CLI applies the catalog on the node you are connected to. Use this method when you are deploying to a single node, when you are working through a rollout node by node, or when you are troubleshooting a change that did not apply as expected.

  1. Connect to the node.
  2. Confirm Puppet is enabled and no run is in progress. This command requires RVC 3.13.0 or later. On earlier versions, use rvc puppet disabled? instead.
rvc puppet status
  1. Run the Puppet agent.
rvc puppet run
  1. Confirm the run completed and review the result. On RVC versions earlier than 3.13.0, review the output of rvc puppet run instead.
rvc puppet status

The following options are available on rvc puppet run and are useful during a deployment.

OptionDescription
--noopSimulates the run without applying changes. See Previewing a change before you deploy it.
--environmentRuns against a different Puppet environment than the one set in the SC//Reliant Platform Manager. Useful for testing a change from a feature branch on a single node before merging it.
--random-waitDelays the start of the run by a random amount of time, up to the given number of seconds. If you leave out this option, the run starts immediately. A value of 0 doesn't mean no delay: RVC applies its default random wait of up to 300 seconds instead. To start a run with effectively no delay, leave out the option or use 1.
--forceRuns Puppet even when it is disabled. Puppet is enabled, the run is started, and Puppet is disabled again afterward. Use this deliberately, since Puppet is usually disabled for a reason.
--clusterCoordinates the run across a cluster. Accepted values are none, primary, and secondary. Requires RVC 3.13.0 or later.

With RVC 3.13.0 or later, Puppet writes to /var/log/puppetlabs/current_execution.log while a run is in progress. When the run completes, the file is renamed to /var/log/puppetlabs/last_execution.log. Check these logs when a deployment does not produce the result you expected. Earlier RVC versions don't write these files, so review the output of rvc puppet run or the RVC server journal with journalctl -u rvc-server.

To stop a run that is in progress, use rvc puppet stop.

For the full command reference, including all arguments, see the rvc puppet pages in the RVC client reference.

Configuring automated Puppet runs

Between the on-demand runs you trigger yourself, the Puppet agent runs automatically on a cron schedule. The properties that control the schedule, and how to check the effective schedule on a node, are described in Configuring automated Puppet runs.

Deciding whether you need maintenance mode or standby

A Puppet run to deploy general configuration changes usually does not require maintenance mode, and it usually does not require placing a node on standby either. Both controls exist to stop Pacemaker from reacting to something you are doing deliberately, and a routine configuration deployment does not normally provoke a reaction worth suppressing. For the full explanation of both controls and how they interact, see Operating the cluster.

Consider whether a control is warranted when your change touches resources that Pacemaker manages. Node group properties such as Cluster KVM Resources, Cluster Docker Resources, Cluster Mount Points, Cluster Virtual IPs, and Cluster Interfaces define cluster-managed resources. When Puppet reconfigures one of these, Pacemaker may see the resource stop and restart, and it may respond by migrating workloads or, in the case of a monitored interface, by acting on what looks like a failure.

The two controls address different situations.

ControlWhen it is worth considering during a deployment
Maintenance modeYour change reconfigures cluster-managed resources and you do not want Pacemaker to fail workloads over while the change is applied. Maintenance mode freezes resource management cluster-wide, so workloads stay where they are. Remember to turn it off afterward, because failover stays disabled while it is on.
StandbyYou want a node emptied before the change is applied to it, such as a rolling deployment where you move workloads off one node, deploy and verify, then return it to service. Standby requires that the cluster is not in maintenance mode, because it relies on Pacemaker to migrate the resources.

Do not combine the two expecting both effects. Placing a node on standby while the cluster is in maintenance mode marks the node standby but leaves its resources stranded on it, which can be mistaken for a completed evacuation. See Operating the cluster.

If you are unsure whether a specific change warrants maintenance mode, contact SC//Reliant Support or your account manager before deploying.

Verifying a deployment

Verify the outcome on the nodes rather than assuming the run succeeded.

  1. Confirm the run finished. With RVC 3.13.0 or later, State should be stopped and Last Run At should reflect your run. On earlier versions, the output of rvc puppet run reports the result.
rvc puppet status
  1. Confirm the change itself is present on the node. What you check depends on the change, such as the state of a container, a virtual machine definition, or the contents of a managed file.
  2. If the change affected a cluster, confirm the cluster is healthy before considering the deployment complete. For more information, see Verifying cluster health.

If a node was offline or unreachable during the deployment, no action is needed to retry it. Puppet enforces the desired state on every run, so the node applies the change on its first successful run after it returns.