PSM sessions fail to connect or connect without recording

TL;DR — When PSM sessions will not connect, the fastest wins are usually: confirm the target port is reachable from the PSM host, confirm the PSM/web gateway can complete the WebSocket/TLS handshake, and confirm the recording path has free disk and writable permissions. If sessions connect but are not recorded, treat it first as a recorder process, filesystem, or object-storage write failure until proven otherwise. Reading time: ~6 min

The scenario

It is Tuesday at 14:40. You just rotated a gateway certificate and rolled a small reverse-proxy change, and now admins say PSM sessions either hang on "connecting" or open briefly with a black screen. A few sessions do start, but the audit team cannot find any recordings for them. The app dashboard looks green, CPU is fine, and the only clue in chat is a screenshot of a browser console showing a failed wss:// request.

Symptoms

• Session launch hangs at a spinner or shows a generic message like:

  Failed to connect to session
  Connection timed out
  Unable to establish secure WebSocket connection
  

• Browser dev tools show WebSocket/TLS errors such as:

  WebSocket connection to 'wss://psm.example.com/session/abc123' failed:
  Error during WebSocket handshake: Unexpected response code: 301
  

  or:

  net::ERR_CERT_COMMON_NAME_INVALID
  net::ERR_SSL_PROTOCOL_ERROR
  

• Reverse proxy logs show upgrade failures:

  10.0.4.21 - - [05/Aug/2026:14:43:11 +0000] "GET /session/abc123 HTTP/1.1" 400 157 "-" "Mozilla/5.0" "-"
  upstream prematurely closed connection while reading response header from upstream
  

• PSM/recorder service logs contain lines like:

  dial tcp 10.20.30.40:3389: connect: no route to host
  dial tcp 10.20.30.40:22: i/o timeout
  permission denied writing /var/lib/psm/recordings/2026/08/05/abc123.rec
  no space left on device
  ffmpeg exited with code 1
  upload failed: 403 Forbidden
  

• Sessions connect, but no recording appears in the audit UI or storage backend.
• Recording files exist but are zero bytes or truncated.
• systemctl shows the recorder/worker unhealthy or restarting.

Likely causes

Step-by-step diagnosis

1. Check whether the PSM host can reach the target system on the session port. Run from the PSM host or the pod/container that actually initiates the target connection:

   nc -vz 10.20.30.40 3389
   nc -vz 10.20.30.41 22
   

   Problem output looks like:

   nc: connect to 10.20.30.40 port 3389 (tcp) failed: No route to host
   nc: connect to 10.20.30.41 port 22 (tcp) failed: Connection timed out
   

   If you get timeout/refused/no route, jump to Fixes → Target host/port unreachable from the PSM host.

2. Test the public/session endpoint for bad redirects or missing WebSocket upgrade support.

   curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" https://psm.example.com/session/test
   

   This is your problem if you see a redirect or plain HTTP response instead of a 101 upgrade, for example:

   HTTP/2 301
   location: /login
   

   or:

   HTTP/1.1 400 Bad Request
   Server: nginx
   

   Jump to Fixes → Reverse proxy/WebSocket misconfiguration or redirect on upgrade path.

3. Validate the certificate actually served on the PSM FQDN.

   openssl s_client -connect psm.example.com:443 -servername psm.example.com </dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
   

   This is your problem if SANs do not include the hostname, the cert is expired, or a different cert is being served after a proxy change. Jump to Fixes → TLS/certificate mismatch on the PSM endpoint.

4. Check recorder service health before digging into storage.

   systemctl --no-pager --full status psm-recorder
   journalctl -u psm-recorder -n 100 --no-pager
   

   This is your problem if you see restart loops, non-zero exits, segfaults, codec errors, or repeated auth/upload failures. Jump to Fixes → Recorder service down/crashing.

5. Check disk space and write permission on the recording path.

   df -h
   sudo -u psm touch /var/lib/psm/recordings/.write-test
   ls -ld /var/lib/psm/recordings
   

   This is your problem if Use% is 100%, touch returns Permission denied, or the directory owner/mode changed. Jump to Fixes → Recording path full or not writable.

6. If recordings are uploaded off-box, test the backend directly. For generic HTTP/S3-compatible endpoints:

   curl -I https://storage.example.com/
   

   Problem output often looks like:

   HTTP/1.1 403 Forbidden
   x-amz-request-id: ...
   

   A 403 can still mean the endpoint is reachable but credentials/policy are wrong; a timeout/DNS failure means network. Jump to Fixes → Object storage/NFS upload target rejecting writes.

7. Check mandatory access control denials if permissions look correct but writes still fail.

   ausearch -m avc -ts recent
   

   If you see denials against the recorder binary or recording path, jump to Fixes → SELinux/AppArmor blocking recorder writes or capture process.

8. Check clock skew if startup tokens or TLS look intermittently invalid.

   timedatectl status
   

   If NTP is off or the clock is materially wrong, jump to Fixes → Time skew breaking tokens/TLS/session startup.

Fixes

Target host/port unreachable from the PSM host

Confirm routing and firewall policy from the PSM node to the target.

ip route get 10.20.30.40
traceroute -T -p 3389 10.20.30.40
sudo nft list ruleset

If the target is behind a host firewall, open the required port there. Examples:

sudo ufw allow from <psm-subnet> to any port 3389 proto tcp
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="<psm-subnet>" port protocol="tcp" port="22" accept'
sudo firewall-cmd --reload

If the issue is a security group/network ACL in your cloud provider, add an inbound rule allowing the PSM source CIDR to the target port, and an outbound rule from the PSM subnet if your egress is restricted.

Verify it worked:

nc -vz 10.20.30.40 3389

Reverse proxy/WebSocket misconfiguration or redirect on upgrade path

For nginx, the session path must preserve upgrade headers and must not redirect authenticated upgrade requests to a login page.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 443 ssl http2;
    server_name psm.example.com;

    location /session/ {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
        proxy_buffering off;
    }
}

Reload safely:

sudo nginx -t && sudo systemctl reload nginx

Trade-off: disabling buffering and increasing timeouts is correct for long-lived interactive sessions, but it increases the impact of idle connections on the proxy.

Verify it worked:

curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" https://psm.example.com/session/test

Look for HTTP/1.1 101 Switching Protocols.

TLS/certificate mismatch on the PSM endpoint

Install the correct certificate chain on the terminating proxy/load balancer and confirm SNI routing is serving the intended cert.

sudo cp fullchain.pem /etc/nginx/certs/psm.example.com.crt
sudo cp privkey.pem /etc/nginx/certs/psm.example.com.key
sudo nginx -t && sudo systemctl reload nginx

If a load balancer terminates TLS, update the cert there instead of on nginx. If both LB and nginx terminate TLS, verify both layers; mismatches are common after partial rotations.

Verify it worked:

openssl s_client -connect psm.example.com:443 -servername psm.example.com </dev/null | openssl x509 -noout -dates -ext subjectAltName

Recorder service down/crashing

Restart it once, then read the last 200 lines before changing anything else.

sudo systemctl restart psm-recorder
systemctl --no-pager --full status psm-recorder
journalctl -u psm-recorder -n 200 --no-pager

If logs show missing binaries or codec helpers, reinstall the package or restore the expected runtime dependency. If it is a container, inspect the crash reason:

docker ps -a --filter name=psm-recorder
docker logs --tail 200 psm-recorder

If the service is OOM-killed, raise memory limits or reduce concurrent captures.

Verify it worked:

systemctl is-active psm-recorder

Expected output:

active

Recording path full or not writable

⚠️ Deleting old recordings can violate retention policy or destroy audit evidence. Confirm your retention requirements before removing anything.

Free space or move the recording path to a larger volume, then restore ownership.

df -h
sudo du -sh /var/lib/psm/recordings/* | sort -h | tail
sudo chown -R psm:psm /var/lib/psm/recordings
sudo chmod 0750 /var/lib/psm/recordings
sudo -u psm touch /var/lib/psm/recordings/.write-test

If you need to move the path:

sudo systemctl stop psm-recorder
sudo rsync -aHAX /var/lib/psm/recordings/ /mnt/recordings/
sudo mv /var/lib/psm/recordings /var/lib/psm/recordings.bak
sudo ln -s /mnt/recordings /var/lib/psm/recordings
sudo systemctl start psm-recorder

Trade-off: symlinks are quick, but a bind mount or direct config change is cleaner long term.

Verify it worked:

sudo -u psm sh -c 'echo test > /var/lib/psm/recordings/.write-test && ls -l /var/lib/psm/recordings/.write-test'

Object storage/NFS upload target rejecting writes

For S3-compatible backends, re-check endpoint, bucket policy, and credentials used by the recorder worker. If the backend is mounted via NFS, verify mount health and write permissions.

mount | grep nfs
showmount -e nfs.example.com
curl -I https://storage.example.com/

If credentials were rotated, update the environment or secret file used by the recorder service, then restart it.

sudo systemctl restart psm-recorder

A 403 Forbidden from object storage usually means bad credentials or policy, not network. A timeout means DNS/routing/firewall.

Verify it worked:

journalctl -u psm-recorder -n 50 --no-pager | grep -Ei 'upload|stored|recording completed'

SELinux/AppArmor blocking recorder writes or capture process

If SELinux is enforcing and the path context is wrong, relabel it.

getenforce
ls -Zd /var/lib/psm/recordings
sudo restorecon -Rv /var/lib/psm/recordings
ausearch -m avc -ts recent

For AppArmor, inspect denials and update the profile if needed.

sudo dmesg | grep -i apparmor

Do not leave SELinux/AppArmor disabled as the fix; use a proper policy update.

Verify it worked:

ausearch -m avc -ts recent | tail

No new denials after a test session is the success signal.

Time skew breaking tokens/TLS/session startup

Re-enable NTP and sync the clock.

timedatectl set-ntp true
timedatectl status

If you run chrony:

chronyc tracking
chronyc sources -v

Large skew can break short-lived session tokens and certificate validation.

Verify it worked:

timedatectl status

Look for System clock synchronized: yes.

Prevention

• Add a synthetic WebSocket probe from outside and inside the network. Example cron probe:

  curl -fsS -o /dev/null -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" https://psm.example.com/session/health | grep -q "101 Switching Protocols"
  

• Alert on recorder disk pressure before it hits failure. Node exporter/Prometheus rule example:

  - alert: PSMRecordingDiskLow
    expr: (node_filesystem_avail_bytes{mountpoint="/var/lib/psm/recordings"} / node_filesystem_size_bytes{mountpoint="/var/lib/psm/recordings"}) < 0.15
    for: 10m
  

• Pin and test reverse-proxy config in CI before deploys:

  nginx -t
  docker run --rm -v $PWD/nginx.conf:/etc/nginx/nginx.conf:ro nginx:stable nginx -t
  

• Add a post-deploy certificate/SAN check:

  openssl s_client -connect psm.example.com:443 -servername psm.example.com </dev/null | openssl x509 -noout -dates -ext subjectAltName | tee /tmp/psm-cert.txt
  grep -q "DNS:psm.example.com" /tmp/psm-cert.txt
  

• Monitor recorder service restarts and non-zero exits:

  systemctl show psm-recorder -p NRestarts -p ExecMainStatus
  

  Export these via your host agent and alert when restarts increase.
• Add a write test for the recording backend in your health checks, using the same service account the recorder uses, not your admin credentials. That catches permission drift and expired secrets before the next real session fails.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.