SSH cert auth fails: fix principals, validity windows, and CA trust

TL;DR — When SSH certificate login fails, the fastest path is: inspect the client cert with ssh-keygen -Lf, confirm the principal matches the target account/host policy, confirm the validity window against the server clock, then verify the server actually trusts the issuing CA via TrustedUserCAKeys (user certs) or @cert-authority / HostCertificate paths as appropriate. The single most common fix is updating the server-side CA trust line or signing the cert with the principal the server expects. Reading time: ~6 min

The scenario

It is Tuesday afternoon, you are trying to hop onto a production box with your usual short-lived SSH certificate, and a deploy window is already open. Your public key still works on a break-glass bastion, but the direct login path that should be enforced by PAM policy is suddenly rejecting you. You did rotate the CA bundle this morning, and someone also changed sshd_config during a hardening pass. Now ssh -vvv is scrolling by with certificate lines, but the session still ends in Permission denied (publickey).

Symptoms

• Client sees one of these at the end of the SSH attempt:

Permission denied (publickey).

Received disconnect from 10.0.12.34 port 22:2: Too many authentication failures
Disconnected from 10.0.12.34 port 22

sign_and_send_pubkey: signing failed for RSA-CERT "/home/alice/.ssh/id_ed25519-cert.pub" from agent: agent refused operation

• ssh -vvv shows the cert being offered, then rejected:

debug1: Offering public key: /home/alice/.ssh/id_ed25519-cert.pub ED25519-CERT SHA256:3x... explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

• Server log on Debian/Ubuntu (journalctl -u ssh or /var/log/auth.log) shows principal or CA issues:

sshd[18422]: Certificate invalid: name is not a listed principal
sshd[18422]: Failed publickey for deploy from 203.0.113.24 port 51522 ssh2: ED25519-CERT ID ops-20261030 (serial 4182) CA ED25519 SHA256:Ab...

sshd[18422]: Certificate invalid: expired

sshd[18422]: Certificate invalid: not yet valid

sshd[18422]: error: Certificate signed by untrusted CA

• Host verification fails after a host cert rollout:

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
Offending RSA key in /home/alice/.ssh/known_hosts:88
Host key verification failed.

• ssh-keygen -Lf on the cert shows unexpected principals or dates:

/home/alice/.ssh/id_ed25519-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:3x...
        Signing CA: ED25519 SHA256:Ab... (using ssh-ed25519)
        Key ID: "ops-20261030"
        Serial: 4182
        Valid: from 20261030120000 to 20261030180000
        Principals:
                alice@example.com
        Critical Options: (none)
        Extensions:
                permit-pty
                permit-port-forwarding

Likely causes

Step-by-step diagnosis

1. Inspect the certificate on the client.

ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub

If you see the wrong Type (host certificate when you are logging in as a user), the wrong Principals, or a Valid: window that does not include current UTC time, this is your problem. Jump to the matching fix under Wrong principal, Expired/not-yet-valid cert, or Wrong cert type.

1. Compare the cert validity window to the server clock, not just your laptop clock.

ssh breakglass@host 'date -u'

If the server time is outside the cert Valid: range, or is obviously skewed by minutes/hours, this is your problem. Jump to Expired or not-yet-valid certificate / clock skew.

1. Confirm the client is offering the cert you expect.

ssh -vvv -i ~/.ssh/id_ed25519 user@host true

What indicates the problem:

debug1: Will attempt key: /home/alice/.ssh/id_ed25519 ED25519 SHA256:... explicit
debug1: Will attempt key: /home/alice/.ssh/id_rsa RSA SHA256:... agent
# but no line offering /home/alice/.ssh/id_ed25519-cert.pub

If the -cert.pub is never offered, the cert file is missing, mismatched to the private key, or IdentitiesOnly yes / agent state is interfering. Jump to Client is not offering the intended cert.

1. Check what the server trusts for user certificates.

sudo sshd -T | grep -Ei 'trustedusercakeys|authorizedprincipalsfile|authorizedprincipalscommand'

If trustedusercakeys is empty, points to a missing file, or points to the wrong CA pubkey, this is your problem. Jump to Server does not trust the issuing CA.

1. Check server logs during a single failed attempt.

sudo journalctl -u ssh -n 50 --no-pager

On some systems:

sudo tail -n 50 /var/log/auth.log

If you see name is not a listed principal, jump to Authorized principals policy rejects the principal. If you see Certificate signed by untrusted CA, jump to Server does not trust the issuing CA. If you see expired or not yet valid, jump to Expired or not-yet-valid certificate / clock skew.

1. If this is host certificate verification, inspect host trust on the client instead of user auth on the server.

ssh-keygen -Lf /etc/ssh/ssh_host_ed25519_key-cert.pub

And check your known hosts CA line:

grep '@cert-authority' ~/.ssh/known_hosts /etc/ssh/ssh_known_hosts 2>/dev/null

If host certs were rolled out but clients only trust raw host keys, or the CA line pattern does not match the hostname, jump to Wrong trust path or wrong cert type.

Fixes

Cert principal does not match what sshd accepts for that account

Inspect what principal is in the cert:

ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub | sed -n '/Principals:/,/Critical Options:/p'

If your server maps principals directly to Unix accounts, sign a new cert with the login name as a principal:

ssh-keygen -s /secure/ca/user_ca -I alice-20261030 -n alice -V +8h ~/.ssh/id_ed25519.pub

If your policy uses role principals and AuthorizedPrincipalsFile, add the principal for that account:

sudo install -d -m 0755 /etc/ssh/auth_principals
printf '%s\n' deploy prod-admin > /tmp/alice.principals
sudo install -m 0644 /tmp/alice.principals /etc/ssh/auth_principals/alice
sudo systemctl reload ssh || sudo systemctl reload sshd

Relevant sshd_config snippet:

TrustedUserCAKeys /etc/ssh/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
PubkeyAuthentication yes

Verify it worked:

ssh -i ~/.ssh/id_ed25519 alice@host 'id -un'

Expired or not-yet-valid certificate / clock skew

Check cert times and UTC now:

ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub | grep '^        Valid:'
date -u
ssh breakglass@host 'date -u'

If the cert is expired, re-sign it with a fresh window:

ssh-keygen -s /secure/ca/user_ca -I alice-20261030-2 -n alice -V +8h ~/.ssh/id_ed25519.pub

If it is not yet valid, either your CA signed with a future start time or a machine clock is wrong. On a systemd host, force NTP sync:

sudo timedatectl set-ntp true
sudo systemctl restart systemd-timesyncd 2>/dev/null || true
timedatectl status

If you need a small skew tolerance during issuance, backdate the start slightly:

ssh-keygen -s /secure/ca/user_ca -I alice-20261030-3 -n alice -V -5m:+8h ~/.ssh/id_ed25519.pub

Trade-off: backdating reduces friction during minor clock skew, but it slightly widens the usable window if a cert leaks. Verify it worked:

ssh -i ~/.ssh/id_ed25519 alice@host 'date -u'

Server does not trust the issuing CA (TrustedUserCAKeys wrong/missing)

Put the correct CA public key on the server and point sshd at it:

sudo install -d -m 0755 /etc/ssh
sudo install -m 0644 /tmp/user_ca.pub /etc/ssh/user_ca.pub
sudo grep -q '^TrustedUserCAKeys ' /etc/ssh/sshd_config && sudo sed -i 's#^TrustedUserCAKeys .*#TrustedUserCAKeys /etc/ssh/user_ca.pub#' /etc/ssh/sshd_config || echo 'TrustedUserCAKeys /etc/ssh/user_ca.pub' | sudo tee -a /etc/ssh/sshd_config
sudo sshd -t
sudo systemctl reload ssh || sudo systemctl reload sshd

Confirm the fingerprint matches the cert issuer:

ssh-keygen -lf /etc/ssh/user_ca.pub
ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub | grep 'Signing CA:'

If you rotated CAs, trust both old and new temporarily by concatenating both public keys into the trust file:

cat old_user_ca.pub new_user_ca.pub | sudo tee /etc/ssh/user_ca.pub >/dev/null
sudo systemctl reload ssh || sudo systemctl reload sshd

Trade-off: dual trust avoids lockouts during rotation, but extends acceptance of the old CA until you remove it. Verify it worked:

sudo sshd -T | grep -i trustedusercakeys
ssh -i ~/.ssh/id_ed25519 alice@host 'true'

Authorized principals policy rejects the principal

Check the active policy source:

sudo sshd -T | grep -Ei 'authorizedprincipalsfile|authorizedprincipalscommand'

If using files, add the principal to the target user’s file:

sudo install -d -m 0755 /etc/ssh/auth_principals
printf '%s\n' alice@example.com alice > /tmp/alice.principals
sudo install -m 0644 /tmp/alice.principals /etc/ssh/auth_principals/alice
sudo systemctl reload ssh || sudo systemctl reload sshd

If using a command, run it manually as the configured user and inspect output/exit code:

sudo -u nobody /usr/local/bin/ssh-authorized-principals alice; echo $?

A non-zero exit or empty output means sshd has nothing to match against. Verify it worked:

ssh -i ~/.ssh/id_ed25519 alice@host 'id -Gn'

Wrong cert type or wrong trust path used

For user login, the cert must say user certificate:

ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub | sed -n '1,6p'

If it says host certificate, issue the correct user cert instead:

ssh-keygen -s /secure/ca/user_ca -I alice-20261030 -n alice -V +8h ~/.ssh/id_ed25519.pub

For host certificate trust on clients, add the CA line to known_hosts:

printf '%s %s\n' '@cert-authority *.example.internal' "$(cat /tmp/host_ca.pub)" >> ~/.ssh/known_hosts

If the hostname is db01.prod.example.internal, a pattern like *.example.internal will not match two labels deep on older/simple patterns; use an explicit domain pattern that matches your naming, or list exact hosts. Verify it worked:

ssh -v alice@db01.prod.example.internal true

Client is not actually offering the intended cert

OpenSSH automatically pairs id_ed25519 with id_ed25519-cert.pub when filenames match. If they do not, either rename them to match or specify both explicitly:

ssh -i ~/.ssh/id_ed25519 -o CertificateFile=~/.ssh/alice-prod-cert.pub alice@host

If the agent is getting in the way, force only the specified identity:

ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 -o CertificateFile=~/.ssh/id_ed25519-cert.pub alice@host

If the private key and cert do not belong together, ssh-keygen output will show different public key fingerprints. Re-issue the cert against the correct public key. Verify it worked:

ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 -o CertificateFile=~/.ssh/id_ed25519-cert.pub alice@host true 2>&1 | grep 'Offering public key:'

Prevention

• Pin CA trust and principals policy in config management, then test syntax before reload:

cat >/etc/ssh/sshd_config.d/50-ca.conf <<'EOF'
TrustedUserCAKeys /etc/ssh/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
PubkeyAuthentication yes
EOF
sshd -t

• Add a CI check that every issued cert has expected principals and a bounded TTL:

ssh-keygen -Lf build/id_ed25519-cert.pub | awk '/Principals:/{flag=1;next}/Critical Options:/{flag=0}flag' | grep -qx '        deploy'
ssh-keygen -Lf build/id_ed25519-cert.pub | grep -Eq 'Valid: from .* to .*'

• Monitor for server log patterns that indicate cert policy drift:

journalctl -u ssh -p warning --since '15 min ago' | grep -E 'untrusted CA|not a listed principal|expired|not yet valid'

Alert on any non-zero match count.

• Check time sync on CA and SSH targets, not just app nodes:

timedatectl show -p NTPSynchronized -p TimeUSec --value

Fail host health checks if NTPSynchronized=no.

• During CA rotation, stage dual trust and add an expiry reminder to remove the old CA:

cat /etc/ssh/user_ca_old.pub /etc/ssh/user_ca_new.pub > /etc/ssh/user_ca.pub
systemctl reload ssh || systemctl reload sshd

Then verify fingerprints in a post-change script:

ssh-keygen -lf /etc/ssh/user_ca.pub

• Add a smoke test from a clean client container so agent state does not hide problems:

docker run --rm -v "$PWD/keys:/keys:ro" alpine:latest sh -lc 'apk add --no-cache openssh-client >/dev/null && ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i /keys/id_ed25519 -o CertificateFile=/keys/id_ed25519-cert.pub alice@host true'

This article was written by an AI system and published pending human review. Verify anything you intend to act on.