Troubleshooting RLSI issues
This page collects the problems you are most likely to meet in the SC//Reliant™ Local Status Indicator (RLSI), grouped by where they show up.
When RLSI does not respond
First establish whether the problem is reaching the node or RLSI itself.
ping <node_hostname_or_ip_address>If the node does not answer, the problem is network connectivity or name resolution rather than RLSI. Confirm you are on a network that routes to the node, and try the IP address in place of the hostname.
If the node answers but the console does not load, and you have shell access, check the service from an SSH session on the node.
-
Ask RLSI for a response locally. This takes the network out of the picture, so it tells you which side the problem is on.
curl -kI https://localhost/A set of response headers means RLSI is up and serving, so the problem is between your computer and the node. Nothing at all means RLSI is not answering.
-
Check the service. RLSI is served by nginx, so check both.
systemctl status rlsi systemctl status nginx -
Restart RLSI if it is not running, or is running but not responding. A
502 Bad Gatewayis usually resolved this way.sudo systemctl restart rlsi.service -
Read the logs if the restart does not resolve it.
journalctl -f -u rlsi.serviceThe nginx logs in
/var/log/nginx/are also worth reading.
If RLSI is not installed on the node at all, see Configuring RLSI.
Reaching the console
| Symptom | Likely cause | What to do |
|---|---|---|
| The browser cannot connect at all | Your network cannot route to the node. | Confirm with ping <node_hostname_or_ip_address>, and check you are on a network with access to the node. |
| The hostname does not resolve | The node's hostname is not resolvable from your network. | Use the node's IP address instead. |
Connection refused | The node is reachable but nothing is listening on the port you targeted. | Work through When RLSI does not respond. If the node uses a non-standard port, target that port. See Setting the listen port. |
The page does not load and you used http:// | RLSI serves HTTPS. | Use https://. |
| A certificate warning you cannot bypass | Some browsers block an untrusted certificate more aggressively than others. | Use a different browser, or bypass the interstitial as your browser allows. |
502 Bad Gateway | RLSI is not responding behind nginx. On the VMS tab specifically, the status refresh interval may be too short for the number of VMs on the node. | Restart the service with sudo systemctl restart rlsi.service. If it recurs on the VMS tab, see Choosing polling intervals. |
| The console loads but the details look wrong for the store | You are connected to a different node than you intended. | Check the store number and role in the header bar, and the Hostname row on the STATUS tab. |
| The console cannot be reached after a configuration change | The listen port changed. | Target the new port. See Setting the listen port. |
Logging in
| Symptom | Likely cause | What to do |
|---|---|---|
| A tab asks for a password unexpectedly | The tab is enabled but not listed as publicly available. | See Hiding the Logs and Admin tabs. |
| The password from your credential file is rejected | The sequence number rotated between reading it and submitting the password. | Reload the login page, read the new number, and retry in one pass. |
| Logins keep failing whatever password you use | The VMS tab's background polling advances the sequence number while you authenticate. | Close the VMS tab while logging in, or raise the VM polling time. See Choosing polling intervals. |
Tabs and checks
| Symptom | Likely cause | What to do |
|---|---|---|
| A tab you expect is missing | For LOGS or ADMIN, it is not listed in Rlsi Enabled Tabs. For VMS or BACKUPS, its section in the configuration hash is off. | See Configuring RLSI. |
| VMS or BACKUPS is still present, or opens with no password, after editing the tab properties | Expected. Those properties do not control either tab. | Turn off its section in the configuration hash instead. |
| The check list is empty, or RLSI errors on load | No checks are enabled. At least one is required. | Add checks to the Rlsi Enabled Checks property. |
| A check you expect is missing | That check is not in the enabled list for this node. | Compare against All available checks. |
| The values look stale | The page reports the values from the last check run. | Reload the page. If Last checked at in the header bar is not advancing either, work through When RLSI does not respond. |
A row reports NOT AVAILABLE | The check produced no usable value, as distinct from not applying to this node. | Escalate with the check name. |
Puppet Ping Status is down | The node cannot reach the Puppet server, so it will drift from its intended configuration. | Check connectivity, then escalate. |
Root File System Usage or Var File System Usage is high | A filesystem is filling. A common cause is the rsyslog spool. | See Reading logs on a node. |
Date is significantly wrong | The node's clock has drifted. | Confirm Network Time Protocol (NTP) is synchronized. A drifted clock also causes false monitoring alerts. |
Internet Gateway Failover Status says the node is on its backup connection | The primary internet connection has failed. | The store is working but has no redundancy. Escalate. See WAN failover. |
Internet Gateway Failover Status reports a stale file or an invalid response | The check could not determine the connection state. | Re-check before acting. Neither message means the connection is down, only that its state is unknown. |
Virtual machines
| Symptom | Likely cause | What to do |
|---|---|---|
An action reports success but the VM never returns to running | The guest operating system is hung, or is slow to start. | Wait and check again. Only after a graceful action has failed, use Hard Reset. |
| The VM list is empty, or a VM you expect is missing | No VMs are defined on this node, or you are on the wrong node. | Check the store number and role in the header bar, then check the other node at the site. |
| The status never updates | RLSI cannot reach the controller on the node. | Escalate. |
Backups and restores
| Symptom | Likely cause | What to do |
|---|---|---|
| Internal server error on the BACKUPS tab | The backup folder configured for the node does not exist. | See The backup target directory. |
| The shutdown countdown runs all the way out during a backup | The VM did not stop, usually because the guest ignored the shutdown request. | Capture the Operation Logs and escalate. |
| The VM is still shut down after a backup finished | The VM did not restart automatically. | Start it from the VMS tab. |
| The cluster is still in maintenance mode after a backup or restore | Maintenance mode was enabled for the operation and has not been turned off. | Escalate — the cluster is not protecting its workloads while it stays on. |
There are no automatic backups to restore from. | No scheduled backups exist for that VM. | Restore from a specific file instead, or take a manual backup first. |
A restore fails on a .qcow2 file | The image may be a "Headless" one, which the form warns can fail. | Escalate with the Operation Logs, and restore from a different backup if one exists. |
Node actions on the Admin tab
| Symptom | Likely cause | What to do |
|---|---|---|
| The node did not come back after a reboot | The reboot is still in progress, or the node failed to start. | RLSI is unreachable while the node is down, so check from the other node at the site. If the node does not return, escalate. |
| The payment container does not come back after the countdown | The container failed to restart. | Escalate. |
| Run Puppet reports errors | The configuration could not be applied, often because required properties are missing for the node. | Read the error, which names the file and line, then escalate. |
| A ping or traceroute fails from the node but works from your workstation | The node's own connectivity is the problem, not the destination. | Check External Ping Status and Routing Status on the STATUS tab. |
| A ping reports the target could not be resolved | DNS resolution is failing on the node. | Ping an IP address directly. If the IP answers and the hostname does not, the problem is DNS rather than connectivity. |
A traceroute shows * * * for a few hops but still reaches the destination | Those routers decline to answer probes. | Nothing to do. This is normal and not a fault. |
A traceroute shows * * * for every hop after a certain point | Traffic is not getting past the last hop that answered. | Note the last hop that answered. If it is hop 1, the problem is at the node's gateway; mid-path it is upstream. |
An interface shows UP BROADCAST MULTICAST but not RUNNING | The interface is enabled but has no link. | Check the physical connection to that port. |
A physical interface shows no inet addr | Expected on this platform. The port is enslaved to a bridge, and the bridge holds the address. | Nothing to do. See Network Configuration. |
| The LAN numbers do not match the interfaces you expect | The LAN-number labels are not reliable on all nodes. | Confirm against View All Interfaces, and against the node's network configuration in the SC//Reliant Platform Manager. |
Configuration changes not taking effect
| Symptom | Likely cause | What to do |
|---|---|---|
| RLSI is not installed after adding the properties | profiles::rlsi is not assigned, or Puppet has not run. | Assign the profile, then deploy and read the run output. |
| A configuration change had no effect, or a change made on the node reverted | Puppet has not run, the run failed, or the file was overwritten because it is Puppet-managed. | Work through Applying a change, making the change as a property rather than on the node. |
Rlsi Version and the installed version disagree | Puppet has not run since the property changed, or the run failed. | See The RLSI version. |
| Puppet errors on a version that does not exist | Rlsi Version names a version the node's repositories do not offer, or the revision suffix is wrong. | Run apt-cache policy rlsi on the node and copy the exact version string. |
Related documentation
- Using the SC//Reliant Local Status Indicator (RLSI): reaching the console, and what to check before acting.
- Configuring RLSI: the properties behind most of the causes above.
- Reading the Status tab in RLSI: what each check reports.
- Managing virtual machines in RLSI: the VM and backup procedures.
- Using the Admin tab in RLSI: the node-level actions.