Using container image signing

The SC//Reliant™ platform can verify that your container images haven't been tampered with and come from a trusted source before they're pulled to your edge devices. Signature validation uses Cosign, the Sigstore project's command-line tool that stores image signatures directly in your container registry alongside the image.

This page describes how to sign your images, add your public key to RPM, and enable signature validation for container pulls across your fleet.

Considerations

• Signature validation applies to both Docker and cluster (singleton/cluster) profiles managed by Puppet.
• If image signature validation fails for one or more images during a Puppet run, only the affected containers (new or updated ones) fail to pull or start. Already running containers are unaffected and continue to run.
• Enabling image signature verification without providing at least one valid public key will cause new container image pulls to fail. Already running containers remain unaffected until they restart or are migrated.
• When signature validation is enabled, every container image pulled through Puppet-managed profiles is validated on each Puppet run. It's not possible to disable signature validation on specific images; enforcement is all or nothing.
• Images pulled from public registries (for example, Docker Hub) that aren't signed with your key will be rejected. If you need to use a public image, pull it into your own registry and sign it with your key first.
• The private key is never installed on edge nodes. Only the public key (configured through RPM) is used at pull time.
• Pushing a signature to the registry requires write access to that registry from your signing machine.

• Signature validation is enforced by a docker wrapper installed at /usr/local/bin/docker, which intercepts docker pull and runs Cosign verification before delegating to the real Docker binary at /usr/bin/docker. Whether a manual docker pull is validated therefore depends on how the shell resolves docker:

  • Interactive shells on a node use the PATH defined in /etc/profile, which places /usr/local/bin ahead of /usr/bin. Manual docker pull commands run from a logged-in shell are intercepted by the wrapper and validated.
  • Non-interactive environments (for example, cron jobs or scripts that don't source /etc/profile) may resolve docker directly to /usr/bin/docker and skip the wrapper. SC//Reliant's docker-image.sh is patched with an explicit PATH export so that Puppet-driven pulls always hit the wrapper regardless of the calling environment.

  If you invoke /usr/bin/docker pull directly, or run docker pull from an environment where /usr/bin precedes /usr/local/bin in PATH, validation is bypassed.

• There is no hard limit on the number of public keys you can add to your tenant in RPM, but we recommend five or fewer to keep image pulls fast.

Behavior summary

The table below summarizes how image signature validation affects containers in different scenarios.

Prerequisites

• Requires the Create and Update a node group capability on your SC//Reliant Platform Manager user role to make the necessary changes to your node group for this feature.
• Must be using profile version 9.4.0 or later. The existing Image Management hash has no signature validation capability.
• A build or CI machine where you can install Cosign and run the signing workflow.
• Write access to your container registry (to push images and their signatures).

Install Cosign

Install Cosign on whichever machine you'll use to sign images. The following examples are for Debian 12. For other platforms, see the Sigstore installation guide.

Option 1: Download the binary

curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64"
sudo mv cosign-linux-amd64 /usr/local/bin/cosign
sudo chmod +x /usr/local/bin/cosign

Option 2: Install the Debian package

LATEST_VERSION=$(curl https://api.github.com/repos/sigstore/cosign/releases/latest | grep tag_name | cut -d : -f2 | tr -d "v\", ")
curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign_${LATEST_VERSION}_amd64.deb"
sudo dpkg -i cosign_${LATEST_VERSION}_amd64.deb

Generate a key pair

Cosign can generate a local key pair with one command. You'll end up with two files:

• cosign.key, your private signing key
• cosign.pub, your public verification key

export COSIGN_PASSWORD=""
cosign generate-key-pair

The private key (plus password, if set) is only needed when signing images. The public key alone is enough for verification. So your edge nodes and the RPM tenant configuration only need the public key.

For production-grade key storage beyond local files, Cosign also supports AWS KMS, GCP KMS, Azure Key Vault, Hashicorp Vault, and Kubernetes Secrets. See the Cosign key management documentation for details.

Sign an image

The example below builds a simple image, pushes it to the SC//Reliant container registry, and signs it. Replace <customer_code> with your tenant's customer code and <image_digest> with the digest returned by docker inspect.

# Build and push
echo "FROM nginx" > Dockerfile
docker build -t registry.reliant.io/clients/<customer_code>/container_registry/cosign-nginx:1 .
docker push registry.reliant.io/clients/<customer_code>/container_registry/cosign-nginx:1
# Get the image digest
docker inspect registry.reliant.io/clients/<customer_code>/container_registry/cosign-nginx:1 | jq -r '.[0].RepoDigests[0]'
# Sign the image by digest
cosign sign --key cosign.key registry.reliant.io/clients/<customer_code>/container_registry/cosign-nginx@sha256:<image_digest>

Cosign pushes the signature as an OCI artifact to the same registry path as the image.

For additional examples and advanced signing patterns, see Signing Containers in the Sigstore documentation.

Add your public key to RPM

1. Open cosign.pub in a text editor. The key is a PEM block that starts with -----BEGIN PUBLIC KEY----- and ends with -----END PUBLIC KEY-----.
2. In RPM, navigate to the Image Management hash for your tenant. If the hash is not present in your node group, add it through the Node Group Properties button.
3. Paste the entire PEM block, including the BEGIN and END lines, into the Signature Public Keys field.
4. To add additional keys, click the Add new value button. To remove a key, click the red − button next to its entry.

Parameters

Enabling Image Signing

Once your public key is in place, complete the following steps to turn on signature validation.

1. Navigate to the node group Hash configuration page.
2. Locate the Image Signing section.
3. Click the Enable this option toggle in the top-right corner of the Image Signing panel so that it turns blue/on.
4. Under Enable Image Signing, select the Enabled radio button.
5. In the Signature Public Keys textarea, paste the public key to be used for image signature verification. If additional public keys are required, click + Add new value and paste each additional key in the new textarea that appears. A maximum of five keys is recommended.
6. Save the configuration using the page's save/apply action.

On the next Puppet run, signature validation becomes active for all container pulls in Docker and cluster profiles.

Disabling Image Signing

1. Navigate to the node group Hash configuration page.
2. Locate the Image Signing section.
3. Click the Enable this option toggle in the top-right corner of the Image Signing panel so that it turns gray/off. Alternatively, leave the section-level toggle enabled but select the Disabled radio button under Enable Image Signing to disable validation while retaining your key configuration.
4. Save the configuration using the page's save/apply action.

Verify an image manually (optional)

Before enabling enforcement across your fleet, you can verify a signed image from any machine that has Cosign installed and access to your public key. Only the public key is required, no password, no private key.

cosign verify --key cosign.pub registry.reliant.io/clients/<customer_code>/container_registry/cosign-nginx@sha256:<image_digest>

A successful verification reports that the Cosign claims were validated, the claims were verified in the transparency log, and the signatures matched the provided public key.

Troubleshooting

Example of a successful verification in journal

Mar 24 13:47:04 <hostname> cosign[670801]: Image registry.reliant.io/clients/<customer_code>/container_registry/cosign-test:1 accepted: signature successfully verified.

Example of a verification failure in journal

Mar 24 13:47:09 <hostname> cosign[671046]: Image registry.reliant.io/clients/<customer_code>/container_registry/cosign-test:1 rejected: signature verification failed.

Example of a Puppet run failure

Error: Execution of 'docker pull <aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/memcached:latest' returned 1: cosign: image rejected - signature verification failed
Error: /Stage[main]/Profiles::Cluster/Profiles::Docker::Images[cluster-docker-images]/Docker_image[<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/memcached:latest]/ensure: change from 'absent' to 'present' failed: Execution of 'docker pull <aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/memcached:latest' returned 1: cosign: image rejected - signature verification failed

Common causes of verification failure

• The public key in RPM doesn't match the private key used to sign the image.
• The image wasn't signed, or was signed after the last Puppet run completed.
• The signature was pushed to a different registry than the image (for example, the image is in ECR but the signature was pushed to a different registry).
• The image was signed by tag and the tag has since been updated. Re-sign the new digest to resolve.
• The image is a public image pulled from a public registry that hasn't been re-signed with your key.

Related documentation

• Image management features
• Using container image staging