Troubleshooting container deployment issues

Deploying your own container to the SC//Reliant™ platform involves building an image, pushing it to a private registry, defining the registry and container in SC//Reliant Platform Manager, and running Puppet on the node. This page covers the problems most commonly encountered at each of those stages, using Amazon Elastic Container Registry (ECR) as the registry.

For the full deployment procedure, see Tutorial: Deploying a custom container to an SC//Reliant node.

Building and pushing the image

These problems occur on your build machine.

Cannot connect to the Docker daemon

Symptom. A docker command fails with an error similar to the following:

Cannot connect to the Docker daemon at unix:///Users/<username>/.docker/run/docker.sock. Is the docker daemon running?

The socket path in the message varies by operating system. On Linux it is typically unix:///var/run/docker.sock.

Cause. The docker command is a client that sends requests to the Docker daemon (the Docker Engine). The client is installed, but the daemon is not running, so there is nothing listening on the socket. On macOS and Windows, the daemon runs inside Docker Desktop, so this error usually means Docker Desktop has not been started since the computer was last restarted, or has been quit.

Resolution.

1. Start the Docker daemon.

   • macOS. Open Docker Desktop from the Applications folder, or run the following:

     open -a Docker

   • Windows. Open Docker Desktop from the Start menu.

   • Linux. Start the Docker service:

     sudo systemctl start docker

2. Wait for the daemon to finish starting. In Docker Desktop, the status in the lower-left corner of the dashboard reads Engine running when it is ready. This can take up to a minute.

3. Confirm that the client can reach the daemon.

   docker version

   The output includes a Server section. If only the Client section is shown, followed by the same error, the daemon is still starting or failed to start.

4. Run the original command again.

If Docker Desktop is running but the error persists, the client might be pointed at a different Docker context. Run docker context ls and confirm that the context marked with an asterisk (*) is desktop-linux. If it is not, run docker context use desktop-linux.

Permission denied under the .docker directory

Symptom. docker build fails with an error similar to the following:

ERROR: stat /Users/<username>/.docker/buildx/refs/desktop-linux/desktop-linux: permission denied

Cause. Files in the .docker directory in your home directory are owned by root rather than by your user account, so your account cannot read or write them. This typically happens when a docker command has previously been run with sudo on the same machine.

Resolution. Run the command again with sudo.

sudo docker build -t <image_name>:<tag> .

If you use sudo for one docker command, use it for every docker command that follows, including signing in to Amazon ECR and pushing the image. Mixing commands run with and without sudo can store credentials and build state where the other commands cannot find them, and can reproduce this error.

Signing in to Amazon ECR fails with 400 Bad Request

Symptom. Signing Docker in to your ECR registry fails with an error similar to the following:

Error response from daemon: login attempt to https://<aws_account_id>.dkr.ecr.<region>.amazonaws.com/v2/ failed with status: 400 Bad Request

Cause. The password is not a valid sign-in token for this registry. The most common reason is that the token was generated for a different AWS Region than the one the repository is in. A token from aws ecr get-login-password is valid only for the registry in the Region it was generated for. If you omit --region, the AWS CLI uses the default Region from your AWS CLI configuration, which might not be the repository's Region. A token that is incomplete, for example because it was only partly copied, causes the same error.

Resolution. Generate the token with --region set to the Region in your registry URI, and sign in again.

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com

The Region in the --region option and the Region in the registry URI must match.

Deploying to the node

These problems appear in the Puppet run on the node after you define the registry and container in SC//Reliant Platform Manager.

Puppet fails to sign in to the registry or pull the image

Symptom. The Puppet run on the node reports one or both of the following errors:

Error response from daemon: login attempt to https://<aws_account_id>.dkr.ecr.<region>.amazonaws.com/v2/ failed with status: 401 Unauthorized

Error: Execution of 'docker pull <aws_account_id>.dkr.ecr.<region>.amazonaws.com/<repository_name>:<tag>' returned 1

Cause. The registry rejected the credentials the node signed in with, so Docker could not pull the image. The access key ID or secret access key entered in the ECR section of the node group is wrong, the key is inactive, or it is a temporary credential. The docker pull error on its own does not include Docker's reason.

Resolution.

1. Check the Access ID and access key in the ECR section of the Cluster Docker Resources or Docker property. They must belong to the same long-term IAM user access key, and the key must be active. If you have only temporary credentials, see Only temporary AWS credentials are available.

2. To see Docker's own error, sign in to the node with SSH and pull the image directly. For more information, see docker.pull.

   rvc docker pull --images=<aws_account_id>.dkr.ecr.<region>.amazonaws.com/<repository_name>:<tag>

3. Correct the settings, save the node group, and run Puppet again. The node signs in again whenever the saved credentials change.

Only temporary AWS credentials are available

Symptom. You have only temporary AWS credentials, such as those issued through AWS IAM Identity Center (SSO), which have an access key ID beginning with ASIA. Entered in the ECR section of the node group, they cause the image pull to fail.

Cause. Temporary credentials require a session token in addition to the access key ID and secret access key. The ECR section has no field for a session token.

Resolution. Ask your AWS administrator for a long-term access key for an IAM user that has permission to pull images from your repository. Its access key ID begins with AKIA. Enter the access key ID and secret access key in the ECR section, save the node group, and run Puppet again.

Accessing the container

These problems occur after the container is running on the node.

Connection refused when connecting to the container

Symptom. The container is running, but connecting to it from another computer fails with an error similar to the following:

curl: (7) Failed to connect to <node_ip_address> port 80 after 142 ms: Couldn't connect to server

Cause. Nothing on the node accepted the connection on that port. The container port is not published on the node, or another container already publishes the same host port, or a Network Access rule does not allow your network.

Resolution.

1. Confirm that you are connecting to the node that is running the container. On a cluster, this is the primary node.

2. Sign in to the node with SSH and check which ports the container publishes.

   sudo docker port docker-container-<container_name>-resource

   If the output is empty, the container port is not published on the node. Set Exposed Ports in the container definition to <host_port>:<container_port>, for example 80:80, save the node group, and run Puppet again.

3. Check that no other container on the node publishes the same host port.

   sudo docker ps --format '{{.Names}}\t{{.Ports}}'

   If another container already publishes the host port, for example 0.0.0.0:80, choose a different host port for your container, for example 8080:80, and connect on that port.

4. If you can connect with curl http://localhost:<host_port> on the node but not from your computer, and the node group has Network Access Control enabled, add a Network Access rule for your computer's network. See Container (Docker) network access.

Related documentation

• Tutorial: Deploying a custom container to an SC//Reliant node: the end-to-end deployment procedure.
• Managing SC//Reliant Platform Manager data for containers: the registry, network, and container settings in the Docker and Cluster Docker Resources properties.
• Container (Docker) network access: Network Access rules for container ports.
• docker.pull: pulling images on a node with RVC.