Troubleshooting RVC jobs

Use this page to find out why an SC//Reliant™ Virtual Controller (RVC) job created with POST api/v2/jobs/rvc did not do what you expected, and to avoid the most common problems. An RVC job can fail at three different points, and each one reports the problem in a different place.

Read the job result

Creating a job only schedules it. Retrieve the job with its payload to see what happened on each node.

ruby api.rb <customer_code> GET "api/jobs/<job_id>?include_payload=true"

Check the following fields in the response:

• status: pending while the job runs and finished when every target node has replied. A finished job is not necessarily a successful one.
• success_nodes_count, failed_nodes_count, and total_nodes_count: how many target nodes succeeded and failed.
• payload: one entry per node hostname. Each entry contains either a result or an error.

The following example shows a job that ran on a node group with two nodes, both of which failed for different reasons.

{
  "id": 1781943,
  "type": "RVC3",
  "status": "finished",
  "message": null,
  "trigged_by": "test.user",
  "success_nodes_count": 0,
  "failed_nodes_count": 2,
  "total_nodes_count": 2,
  "parameters": {
    "method": "cluster.maintenance_mode?",
    "params": {},
    "hostnames": [],
    "update_rpm": true,
    "node_group_id": 2392
  },
  "time_started": "2026-10-11T20:39:56Z",
  "time_finished": "2026-10-11T20:40:06Z",
  "payload": {
    "sit0001red0": [
      {
        "error": {
          "code": 1,
          "data": "This is an RVC agent error. The call did not reach the RVC server.",
          "message": "Error connecting to RVC server."
        },
        "timestamp": 1791751207
      }
    ],
    "sit0001red1": [
      {
        "error": {
          "message": "Node sit0001red1 not discovered by RJE broker"
        },
        "timestamp": 1791751207
      }
    ]
  }
}

Request errors

These errors are returned when the job is created. No job is created and nothing runs on any node.

The API does not check the method name itself when the job is created. A misspelled method in an allowed module, such as sys.bogus, creates a job that then fails on each node with -32601.

Node errors

These errors appear in the error object for a node in the job payload. Errors with a negative code are returned by the RVC server on the node. Errors without a code, or with code 1, are returned before the call reaches the RVC server.

Methods that run in the background

Some methods start a long-running operation and return immediately. Their job finishes with a success status and a short message, such as Creating snapshot for kvm1... or Starting full backup for kvm1, while the operation continues on the node. A successful job for one of these methods only means that the operation was started. If the operation fails later, the job result does not change.

Methods that accept a notify_uri parameter can run in the background. In testing, the following methods returned before their operation completed:

• kvm.snapshot
• kvm.full_backup
• docker.pull
• kvm.start and kvm.shutdown
• kvm.snapshot_restore
• kvm.restore

To confirm that a background operation completed, use one of the following approaches:

• Check the result with a read-only method. For example, run kvm.snapshot_list after kvm.snapshot, kvm.status after kvm.start or kvm.shutdown, and docker.images after docker.pull. Allow time for the operation to finish before you check.
• Receive progress messages. Pass a notify_uri that you control, or subscribe to the module and topic with rvc.subscribe. RVC sends progress messages to the URI, including a final message with the exit code of the operation.
• Wait until the virtual machine is no longer busy. Run rvc.busy until the virtual machine no longer appears in the result. An empty list means no tracked operation is running on the node. rvc.busy tracks the operations started by kvm.start, kvm.full_backup, kvm.restore, kvm.snapshot, and kvm.snapshot_restore, but not kvm.shutdown or docker.pull, so check those with a read-only method instead. For an example of the result, see RVC module examples.
• Read the RVC log on the node. See the next section.

Find the cause in the RVC log on a node

The RVC server on each node writes every request, response, progress message, and error to the system journal. If you have shell access to the node, this is the most direct way to find out why an operation failed.

Search the journal for the time the job ran. Journal times are in the node's local time zone, while job timestamps are in UTC.

sudo journalctl --since "<start_time>" --until "<end_time>" --no-pager | grep -iE "rvc|<vm_name>"

The following entries are the most useful:

For example, the following entries show a full backup that was accepted but then failed because filename was given as a full path:

INFO rvc.messenger: [kvm.kvm1]: Creating /mnt/kvm_backups/mnt/kvm_backups/kvm1.tar.gz file...
ERROR rvc.messenger: [kvm.kvm1]: Aborting full backup. Reason: Error creating full backup file: tar (child): /mnt/kvm_backups/mnt/kvm_backups/kvm1.tar.gz: Cannot open: No such file or directory
INFO rvc.messenger: Final message sent to topic 'kvm1' on module 'kvm' with exit code 1.

Check a node with the RVC module

The following read-only methods help you check a node before or after a job. Run them as RVC jobs like any other method.

Avoid common problems

• Test on one node first. Run a new method against a single hostname and check the payload before you target a node group or saved filter.
• Check the method with rvc.api. Method names and parameters can differ between RVC versions. For example, kvm.destroy and kvm.reset are not available in RVC 3.13.0.
• Use the method names from rvc.api. RVC accepts hyphens in place of underscores, so kvm.snapshot-list and kvm.snapshot_list both work, but the names returned by rvc.api use underscores.
• Do not treat a finished job as a finished operation. For methods that run in the background, confirm the result separately. See Methods that run in the background.
• Give kvm.full_backup a file name, not a path. The archive is written to the directory in the target parameter, or to the full_backup directory in the RVC configuration file if target is not set. A full path in filename is accepted, but the backup fails. In contrast, kvm.full_backup_info needs the full path to the archive.
• Plan for downtime during a full backup. kvm.full_backup shuts the virtual machine down while it copies the disk, and starts it again before it creates the archive. This happens even if the backup later fails.
• Check maintenance mode after restoring a snapshot. In RVC 3.13.0, kvm.snapshot_restore puts the cluster in maintenance mode and does not take it out of maintenance mode when it finishes. While the cluster is in maintenance mode, no cluster resources fail over. After the restore, run cluster.maintenance_mode? and, if it returns true, run cluster.disable_maintenance_mode. Other methods, such as kvm.start, kvm.shutdown, and kvm.restore, take the cluster out of maintenance mode when they finish.
• Wait for one operation to finish before starting the next. Operations on the same virtual machine cannot overlap and fail with -32085. Wait until rvc.busy returns an empty list.
• Do not use kvm.full_backup_info to check whether a backup has finished. It reads only the first members of the archive, and the disk is the last member, so it can succeed while the disk is still being written. Wait for the final progress message with exit code 0, or confirm in the RVC log on the node, before you restore from or copy a new archive.
• Be careful with optimize. Setting optimize to true in kvm.full_backup removes all snapshots of the virtual machine.
• Consolidating snapshots needs at least two snapshots. kvm.snapshot_consolidate merges the oldest snapshots into the latest one. With a single snapshot, it returns Nothing to consolidate.
• Expect offline nodes in large targets. A node group or saved filter can include nodes that are offline. They fail with not discovered by RJE broker without affecting the other nodes. Check failed_nodes_count and the payload for each node.
• Quote URLs on the command line. When you retrieve a job with ruby api.rb, put the path in double quotes, for example "api/jobs/<job_id>?include_payload=true". Some shells, such as zsh, otherwise treat the ? as a wildcard.