Sign OCI images with Cosign and verify before Kubernetes deploy

TL;DR — Sign the image by digest, not by mutable tag, with cosign sign --key cosign.key REGISTRY/IMAGE@sha256:..., then verify the same digest with cosign verify. If you deploy to Kubernetes, enforce verification with an admission policy such as Kyverno verifyImages; the most common failure is signing :latest and then verifying a different digest after the tag moved. Reading time: ~5 min

Goal

When you are done, your container image exists in the registry with a verifiable Cosign signature, cosign verify succeeds for the exact image digest you deploy, and your Kubernetes cluster rejects deployments that reference images without a matching valid signature.

Prerequisites

• Registry push access to your image repository, for example ghcr.io/org/app or registry.example.com/team/app
• Kubernetes cluster admin or policy-admin access if you want enforcement before deployment
• docker or another OCI-compatible builder installed — check with docker --version
• cosign installed — check with cosign version
• kubectl installed — check with kubectl version --client
• Optional but recommended: jq installed for extracting digests — check with jq --version
• A place to store the Cosign private key securely in CI, for example your CI secret store
• The exact image name you will deploy, and a unique tag such as the Git SHA, not only latest

Steps

Step 1: Install Cosign and generate a key pair

cosign version
cosign generate-key-pair
ls -l cosign.key cosign.pub

When this succeeds, you will have cosign.key and cosign.pub in the current directory, and Cosign prompts for a password for the private key.

Step 2: Build and push the image, then capture its digest

export IMAGE=ghcr.io/acme/api
export TAG=$(git rev-parse --short=12 HEAD)

docker build -t ${IMAGE}:${TAG} .
docker push ${IMAGE}:${TAG}
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${IMAGE}:${TAG} | sed 's/.*@//')
echo ${DIGEST}

When this succeeds, echo ${DIGEST} prints a sha256:... value.

If docker inspect does not show RepoDigests, use the registry tool you already have. With crane it is:

crane digest ${IMAGE}:${TAG}

Step 3: Sign the pushed image by digest

export COSIGN_PASSWORD='your-key-password'
cosign sign --key cosign.key ${IMAGE}@${DIGEST}

When this succeeds, Cosign prints output shaped like this:

Pushing signature to: ghcr.io/acme/api

Do not sign :${TAG} if your deployment later resolves a different digest. Sign and verify @sha256:....

Step 4: Verify the signature locally before wiring it into CI/CD

cosign verify --key cosign.pub ${IMAGE}@${DIGEST}

When this succeeds, Cosign returns exit code 0 and prints JSON payloads shaped like this:

[
  {
    "critical": {
      "identity": {"docker-reference": "ghcr.io/acme/api"},
      "image": {"docker-manifest-digest": "sha256:8f..."},
      "type": "cosign container image signature"
    },
    "optional": null
  }
]

A failure looks like this and exits non-zero:

Error: no matching signatures:

main.go:74: error during command execution: no matching signatures

Step 5: Put signing into CI so every pushed image is signed

Store cosign.key, cosign.pub, and the key password in your CI secret store, then run these exact commands after the image push step.

export IMAGE=ghcr.io/acme/api
export TAG=${GIT_SHA}
printf '%s' "$COSIGN_KEY_B64" | base64 -d > cosign.key
printf '%s' "$COSIGN_PUB_B64" | base64 -d > cosign.pub
export COSIGN_PASSWORD="$COSIGN_PASSWORD_SECRET"

docker build -t ${IMAGE}:${TAG} .
docker push ${IMAGE}:${TAG}
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${IMAGE}:${TAG} | sed 's/.*@//')
cosign sign --key cosign.key ${IMAGE}@${DIGEST}
cosign verify --key cosign.pub ${IMAGE}@${DIGEST}

When this succeeds, the job exits 0; if signing or verification fails, the pipeline stops before deployment.

⚠️ If you enforce admission before your workloads are signed, new deployments will be rejected. Apply the policy only after you have verified at least one signed image path end-to-end.

Step 6: Enforce verification in Kubernetes with Kyverno

Install Kyverno if it is not already present:

kubectl create namespace kyverno
kubectl create -f https://github.com/kyverno/kyverno/releases/latest/download/install.yaml
kubectl -n kyverno rollout status deploy/kyverno-admission-controller --timeout=120s

Apply a policy that verifies images from your registry using your public key:

cat <<'YAML' | kubectl apply -f -
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-signed-images
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-acme-api
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/acme/*"
          mutateDigest: false
          required: true
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      REPLACE_WITH_CONTENTS_OF_cosign.pub
                      -----END PUBLIC KEY-----
YAML

When this succeeds, kubectl apply reports clusterpolicy.kyverno.io/verify-signed-images created or configured.

Step 7: Deploy using the signed digest, not a tag

kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -
cat <<YAML | kubectl -n demo apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: ${IMAGE}@${DIGEST}
          imagePullPolicy: IfNotPresent
YAML

When this succeeds, deployment.apps/api created or configured is printed, and the Pod is admitted.

[CI] ---- 1. build/push ----> [Registry]
[CI] ---- 2. sign digest ----> [Registry]
[Deployer] ---- 3. apply Pod ----> [API Server]
[Kyverno] <---- 4. admission ---- [API Server]
[Kyverno] ---- 5. verify sig ----> [Registry]
[API Server] ---- 6. allow/deny ----> [Pod]

Verify it works

Run all three checks.

1. Verify the signature directly:

cosign verify --key cosign.pub ${IMAGE}@${DIGEST}
echo $?

Expected: JSON output and then 0.

1. Confirm the cluster accepts the signed image:

kubectl -n demo rollout status deploy/api --timeout=120s
kubectl -n demo get pods -o wide

Expected: deployment "api" successfully rolled out and a Pod in Running state.

1. Confirm the cluster rejects an unsigned image. Use a different image that is not signed by your key:

kubectl -n demo set image deploy/api api=ghcr.io/acme/unsigned:latest
kubectl -n demo describe deploy/api | sed -n '/Events/,$p'

Expected: the update is denied. The event or API error is shaped like this:

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:

resource Deployment/demo/api was blocked due to the following policies

verify-signed-images:
  verify-acme-api: image ghcr.io/acme/unsigned:latest failed verification

Common pitfalls

Signing a tag and deploying a different digest

Mistake: you run cosign sign ghcr.io/acme/api:latest, then the tag is moved by a later push.

Symptom: cosign verify --key cosign.pub ghcr.io/acme/api@sha256:... returns no matching signatures, or Kubernetes rejects the deployment even though you "signed the image".

Fix: sign and deploy the immutable digest with cosign sign --key cosign.key ${IMAGE}@${DIGEST}.

Public key pasted incorrectly into the policy

Mistake: the Kyverno policy contains extra spaces, missing header lines, or a truncated cosign.pub.

Symptom: admission denies every matching image, including correctly signed ones.

Fix: replace the publicKeys block with the exact contents of cosign.pub, including -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY-----.

CI signs before the image is actually pushed

Mistake: the pipeline computes a local image digest and signs before the registry has the manifest.

Symptom: Cosign errors with registry lookup failures or verification fails against the registry copy.

Fix: run docker push first, then extract the pushed digest from RepoDigests or crane digest, then sign that digest.

Admission policy applied cluster-wide too early

Mistake: validationFailureAction: Enforce is enabled before existing workloads or sidecar images are signed.

Symptom: deploys start failing across namespaces, including unrelated apps.

Fix: narrow imageReferences first, or switch temporarily to Audit, sign the affected images, then change back to Enforce.

Registry auth works for nodes but not for verification

Mistake: the cluster can pull private images, but the admission controller cannot read signature artifacts from the registry.

Symptom: Pods fail admission with verification errors only for private repositories.

Fix: provide registry credentials to the verifier according to your cluster policy engine setup, or test first with a public repository to separate auth issues from signature issues.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.