App fails to connect after password rotation: diagnose and fix fast
For developers handling incidents where an app loses access to a database, SMTP server, API, or other dependency right after automated credential rotation. This runbook gives you a fast decision path, exact checks, and concrete fixes for the common PAM-related failure modes.
TL;DR — If an application stops connecting right after automated password rotation, the most likely issue is that the app is still using the old secret or reading the new secret from the wrong place. Check the effective runtime secret first, then verify the new credential directly against the target service, and only after that chase cache, pool, or replication edge cases. Reading time: ~6 min
The scenario
It is 2:17 PM on a Tuesday. Your password rotation job ran at 2:00 PM, the compliance dashboard is green, and fifteen minutes later your app starts throwing connection errors to Postgres and SMTP. The deploy pipeline is quiet, CPU and memory look normal, and the only thing that changed was the rotated credential. Users are seeing failed logins, delayed emails, or generic 500s, and you need to prove whether the break is in the app, the secret distribution path, or the target service itself.
Symptoms
- App logs start failing immediately after the rotation timestamp.
- Typical database errors:
FATAL: password authentication failed for user "app_user"
psql: error: connection to server at "db.internal" (10.0.12.14), port 5432 failed: FATAL: password authentication failed for user "app_user"
SQLSTATE[08006] [7] connection to server failed
- Typical MySQL errors:
ERROR 1045 (28000): Access denied for user 'app_user'@'10.0.4.22' (using password: YES)
- Typical Redis errors:
NOAUTH Authentication required.
WRONGPASS invalid username-password pair or user is disabled.
- Typical SMTP errors:
535 5.7.8 Authentication credentials invalid
454 4.7.0 Temporary authentication failure
- App health endpoint may still return 200 if it does not check downstream dependencies.
- Existing traffic may partially work while new connections fail, especially with connection pools.
- Restarting one app instance sometimes "fixes" it temporarily if instances are inconsistent.
Likely causes
| Cause | How common | Quick check |
|---|---|---|
| App is still using the old secret in process env, mounted file, or config cache | Very common | ```bash |
| kubectl exec deploy/myapp -- printenv | grep -E 'DB_PASSWORD | SMTP_PASSWORD' |
| Rotation updated the secret store, but the app reads a different key/version/path | Very common | ```bash
grep -R "DB_PASSWORD\|secret/" ./deploy ./infra ./docker-compose.yml
``` |
| Target account password changed, but rotation did not update all replicas/nodes/services | Common | ```bash
PGPASSWORD='NEWPASS' psql -h db.internal -U app_user -d appdb -c 'select 1;'
``` |
| Connection pool or long-lived workers kept old credentials and need recycle | Common | ```bash
ps -ef | grep -E 'gunicorn|sidekiq|celery|puma|node'
``` |
| Username/host binding changed with rotation policy, not just password | Occasional | ```bash
mysql -h db.internal -u app_user -p -e "SELECT CURRENT_USER();"
``` |
| Special characters were rotated in and then mangled by shell, YAML, or URL encoding | Occasional | ```bash
python3 - <<'PY'
import os
print(repr(os.getenv('DATABASE_URL')))
PY
``` |
| Secret volume/env refresh lagged on one node/pod/instance | Occasional | ```bash
kubectl get pods -o wide
``` |
## Step-by-step diagnosis
1. Check whether the incident lines up exactly with rotation.
```bash
# app logs around the event
journalctl -u myapp --since "2026-10-30 13:50" --until "2026-10-30 14:20" | tail -n 50
# or in Kubernetes
kubectl logs deploy/myapp --since=30m | tail -n 100
This is your problem if errors start within minutes of the rotation job and were absent before. If yes, continue to step 2.
- Inspect the effective secret inside a running app process.
kubectl exec deploy/myapp -- sh -lc 'printenv | grep -E "DB_HOST|DB_USER|DB_PASSWORD|DATABASE_URL|SMTP_PASSWORD"'
Or on a VM:
tr '\0' '\n' < /proc/$(pgrep -f myapp | head -n1)/environ | grep -E 'DB_|DATABASE_URL|SMTP_'
This is your problem if the password value, secret path, or version is clearly old or points to the wrong key. Jump to Fixes → App still using old secret or Fixes → App reads wrong key/version/path.
- Test the new credential directly against the target service, outside the app.
PGPASSWORD='NEWPASS' psql -h db.internal -U app_user -d appdb -c 'select 1;'
Typical good output:
?column?
----------
1
(1 row)
Typical bad output:
psql: error: connection to server at "db.internal" (10.0.12.14), port 5432 failed: FATAL: password authentication failed for user "app_user"
If the new password fails here, the rotation likely did not apply correctly on the target service or applied only on some nodes. Jump to Fixes → Target service not updated everywhere.
- Compare old vs new behavior across instances.
kubectl get pods -l app=myapp -o name | xargs -I{} sh -c 'echo === {}; kubectl exec {} -- sh -lc "printenv | grep DB_PASSWORD | sed s/=.*/=REDACTED/"'
This is your problem if some pods have refreshed and others have not. Jump to Fixes → Secret refresh lag on some instances.
- Check whether only new connections fail while old ones still work.
# Postgres app-side symptom: existing pooled sessions survive, new ones fail
kubectl logs deploy/myapp --since=15m | grep -E 'password authentication failed|too many clients|connection reset' | tail -n 50
This is your problem if traffic is partially working and errors spike only when workers recycle or scale up. Jump to Fixes → Connection pool or workers need recycle.
- Inspect for quoting/encoding breakage.
python3 - <<'PY'
import os, urllib.parse
u=os.getenv('DATABASE_URL','')
print('raw=',repr(u))
print('parsed password=',repr(urllib.parse.urlparse(u).password))
PY
This is your problem if the parsed password is truncated, empty, or differs from the rotated secret because of @, :, /, %, #, or $. Jump to Fixes → Special characters mangled.
- Check whether the account identity changed in a way the app does not match.
# MySQL example
mysql -h db.internal -u app_user -p -e "SELECT CURRENT_USER();"
This is your problem if the effective user/host binding is not what the app connects as, or if the rotation created a new user but the app still uses the old username. Jump to Fixes → Username or host binding changed.
Fixes
App still using old secret in env, file, or config cache
Update the runtime secret source, then restart processes that only read secrets at startup.
# Kubernetes secret update example
kubectl create secret generic myapp-secrets --from-literal=DB_PASSWORD='NEWPASS' --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deploy/myapp
kubectl rollout status deploy/myapp
If your app caches config at build/start time, clear it before restart. Examples:
# Laravel
php artisan config:clear
# Symfony
php bin/console cache:clear --env=prod
Verify it worked:
kubectl exec deploy/myapp -- sh -lc 'printenv | grep DB_PASSWORD | sed s/=.*/=REDACTED/'
App reads wrong key/version/path
Fix the reference, not just the secret value. Common break: rotation wrote secret/prod/db/password, app still reads secret/prod/database/password or a pinned version.
# deployment snippet
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: myapp-secrets
key: DB_PASSWORD
If you use a mounted file, inspect and correct the path:
kubectl exec deploy/myapp -- sh -lc 'ls -l /run/secrets && sed -n "1p" /run/secrets/DB_PASSWORD'
Verify it worked:
kubectl exec deploy/myapp -- sh -lc 'test -s /run/secrets/DB_PASSWORD && echo ok'
Target service not updated everywhere
Apply the new password on the actual backend the app reaches. For Postgres:
ALTER ROLE app_user WITH PASSWORD 'NEWPASS';
For MySQL:
ALTER USER 'app_user'@'%' IDENTIFIED BY 'NEWPASS';
FLUSH PRIVILEGES;
If you have replicas, proxies, or failover nodes, test each endpoint the app might hit.
for h in db-primary.internal db-replica1.internal db-replica2.internal; do echo === $h; PGPASSWORD='NEWPASS' psql -h $h -U app_user -d appdb -c 'select 1;'; done
⚠️ Changing credentials on the wrong node or only on a replica can cause immediate write failures or split behavior during failover. Verify it worked:
PGPASSWORD='NEWPASS' psql -h db.internal -U app_user -d appdb -c 'select current_user;'
Connection pool or workers need recycle
If the app only reads credentials at startup, rotate the pool/workers after updating the secret.
# Kubernetes
kubectl rollout restart deploy/myapp
# systemd
sudo systemctl restart myapp
# gunicorn example
pkill -HUP gunicorn || sudo systemctl restart myapp
Trade-off: a rolling restart is safer than a hard restart, but slower to drain old pooled connections. Verify it worked:
kubectl logs deploy/myapp --since=5m | grep -E 'password authentication failed' || echo clean
Username or host binding changed
Some rotations create a new principal or tighten host matching. Update both username and password in app config, or restore the expected host binding.
-- MySQL: inspect current grants
SHOW GRANTS FOR 'app_user'@'%';
-- If needed
CREATE USER 'app_user'@'10.%' IDENTIFIED BY 'NEWPASS';
GRANT SELECT, INSERT, UPDATE, DELETE ON appdb.* TO 'app_user'@'10.%';
Verify it worked:
mysql -h db.internal -u app_user -p -e "SELECT CURRENT_USER();"
Special characters mangled by shell, YAML, or URL encoding
If the rotated password contains reserved characters, stop embedding it raw in a URL unless you URL-encode it.
python3 - <<'PY'
import urllib.parse
pw='p@ss:w/or#d%$'
print(urllib.parse.quote(pw, safe=''))
PY
Example DSN fix:
export DATABASE_URL='postgresql://app_user:p%40ss%3Aw%2For%23d%25%24@db.internal:5432/appdb'
Safer pattern: pass password separately.
export DB_HOST=db.internal
export DB_USER=app_user
export DB_PASSWORD='p@ss:w/or#d%$'
Verify it worked:
python3 - <<'PY'
import os, urllib.parse
print(repr(urllib.parse.urlparse(os.getenv('DATABASE_URL','')).password))
PY
Secret refresh lag on some instances
Force convergence across nodes/pods and confirm all instances are on the same revision.
kubectl rollout restart deploy/myapp
kubectl rollout status deploy/myapp
kubectl get pods -l app=myapp -o wide
If this is on VMs with local secret agents, restart the agent and the app in that order.
sudo systemctl restart secret-agent
sudo systemctl restart myapp
Verify it worked:
kubectl get pods -l app=myapp -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount
Prevention
- Add a post-rotation synthetic login that uses the newly rotated credential against the real service before marking rotation successful.
PGPASSWORD="$ROTATED_DB_PASSWORD" psql -h db.internal -U app_user -d appdb -c 'select 1;' >/dev/null
- Alert on auth failures by dependency, not just app 500s.
# example log pattern for alerting
password authentication failed|Access denied for user|535 5.7.8|WRONGPASS
- Keep app config on discrete vars instead of raw URLs when possible; this avoids URL-encoding bugs during rotation.
DB_HOST=db.internal
DB_PORT=5432
DB_USER=app_user
DB_PASSWORD=...
DB_NAME=appdb
- In CI, validate that deployment manifests reference the intended secret key/path exactly once.
grep -R "DB_PASSWORD\|DATABASE_URL" ./deploy ./infra
- For pooled apps, wire rotation hooks to a rolling restart after secret update.
kubectl rollout restart deploy/myapp && kubectl rollout status deploy/myapp
- Record both secret version and app revision in logs/metrics so you can correlate breakage fast.
{"app_revision":"a1b2c3d","secret_version":"db-pass-v42","msg":"startup complete"}
This article was written by an AI system and published pending human review. Verify anything you intend to act on.
Have a project in mind?
Get an instant AI price estimate for it, or talk directly to our team.
One email a month on what we learn building with AI