Site-to-site VPN tunnels keep dropping: diagnose IKE and MTU issues

TL;DR — If a site-to-site VPN comes up and then drops, or only small packets work, check two things first: IKE/IPsec proposal and lifetime mismatches, and MTU/fragmentation on the path. The fastest wins are usually aligning Phase 1/2 settings on both peers and clamping TCP MSS on the tunnel path. Reading time: ~6 min

The scenario

It is Tuesday at 14:37. Your app deploy is fine, health checks are green, but the private link between your app VPC and the customer network keeps flapping every few minutes. Database connections over the VPN reset mid-query, ping works for tiny payloads, and larger HTTPS requests hang until they time out. The network team says "the tunnel is up on our side," your logs say otherwise, and you need to prove whether this is IKE negotiation, rekey, or plain old MTU.

Symptoms

• Tunnel establishes, then drops on rekey intervals such as every 30 or 60 minutes.
• Only some traffic works: SSH connects, but file transfers, HTTPS POSTs, or DB queries hang.
• ipsec statusall or equivalent shows repeated CHILD_SA creation/deletion.
• Logs contain messages like:

charon: 14[IKE] received NO_PROPOSAL_CHOSEN notify, no CHILD_SA built
charon: 09[IKE] peer did not respond to retransmit 5 of request message ID 0
charon: 11[NET] sending packet: from 203.0.113.10[500] to 198.51.100.20[500]
charon: 11[NET] sending packet: from 203.0.113.10[4500] to 198.51.100.20[4500]
charon: 12[IKE] establishing CHILD_SA net-net{7} failed
charon: 13[ENC] parsed INFORMATIONAL response 3 [ INVALID_SELECTORS ]

• Kernel or firewall counters show encapsulation but no decapsulation, or vice versa.
• ping with DF set fails above a certain size:

$ ping -M do -s 1472 10.20.30.40
PING 10.20.30.40 (10.20.30.40) 1472(1500) bytes of data.
ping: local error: message too long, mtu=1438

• Users see intermittent app errors over private connectivity:

curl: (28) Operation timed out after 30001 milliseconds with 0 bytes received
psql: error: connection to server at "10.20.30.15", port 5432 failed: SSL SYSCALL error: EOF detected
ssh: packet_write_wait: Connection to 10.20.30.22 port 22: Broken pipe

Likely causes

Step-by-step diagnosis

1. Check whether the tunnel is failing at negotiation or after it comes up.

sudo ipsec statusall
sudo journalctl -u strongswan -n 200 --no-pager

This is your problem if you see NO_PROPOSAL_CHOSEN, INVALID_SELECTORS, or repeated establishing CHILD_SA failed. Jump to Fixes → Phase 1/2 proposal mismatch or lifetime/rekey mismatch or Fixes → Traffic selector / subnet mismatch.

1. Confirm the actual packets on the wire before changing config.

sudo tcpdump -ni any 'host <peer-public-ip> and (udp port 500 or udp port 4500 or esp)'

This is your problem if you only see outbound IKE packets and no replies, or replies stop after NAT-T switches to UDP 4500. Jump to Fixes → Firewall/NAT blocking UDP 500/4500 or ESP, especially after idle timeout.

1. Test PMTU with DF set across the private IP path.

ping -M do -s 1372 <remote-private-ip>
ping -M do -s 1410 <remote-private-ip>
ping -M do -s 1460 <remote-private-ip>

This is your problem if smaller sizes work and larger ones fail, or you get message too long / silent loss. Jump to Fixes → Path MTU / fragmentation issue inside ESP or UDP 4500.

1. Verify the installed traffic selectors and routes match what you think the tunnel should carry.

sudo ip xfrm policy
ip route get <remote-private-ip>

This is your problem if the policy subnets do not include the affected source/destination ranges, or traffic routes outside the tunnel. Jump to Fixes → Traffic selector / subnet mismatch.

1. Check whether the tunnel dies on idle or exactly at DPD/rekey intervals.

sudo journalctl -u strongswan --since '2 hours ago' --no-pager | egrep 'DPD|liveness|rekey|deleting IKE_SA|CHILD_SA'

This is your problem if disconnects line up with DPD timeout, liveness check failed, or every rekey event. Jump to Fixes → DPD/keepalive too aggressive or asymmetric or back to the proposal/lifetime fix.

1. Rule out clock skew before chasing ghosts.

timedatectl status
chronyc tracking || ntpq -p

This is your problem if the system clock is significantly off, NTP is unsynced, or logs show cert validity/auth anomalies around tunnel setup. Jump to Fixes → Clock skew causing IKE auth/rekey weirdness.

[Local GW] ---- 1. IKE_SA_INIT UDP 500 ----> [Remote GW]
    |
    | 2. NAT detected, switch to UDP 4500
    v
[ESP-in-UDP] ---- 3. Encrypted data ----> [Path]
    |
    | 4. Large packet + DF + overhead = black hole
    v
[App traffic] <---- 5. Small packets work ---- [Remote LAN]

Fixes

Phase 1/2 proposal mismatch or lifetime/rekey mismatch

Align IKE and ESP proposals, PFS group, lifetimes, and rekey behavior on both peers. On strongSwan, inspect the live config first:

sudo swanctl --list-conns
sudo swanctl --list-sas

Example swanctl.conf fragment:

connections {
  net-net {
    version = 2
    local_addrs = 203.0.113.10
    remote_addrs = 198.51.100.20
    proposals = aes256-sha256-modp2048
    rekey_time = 50m
    dpd_delay = 30s
    dpd_timeout = 120s
    local {
      auth = psk
      id = 203.0.113.10
    }
    remote {
      auth = psk
      id = 198.51.100.20
    }
    children {
      net-net {
        local_ts = 10.10.0.0/16
        remote_ts = 10.20.0.0/16
        esp_proposals = aes256-sha256-modp2048
        life_time = 1h
        rekey_time = 50m
        start_action = trap
      }
    }
  }
}
secrets {
  ike-psk {
    id-1 = 203.0.113.10
    id-2 = 198.51.100.20
    secret = "replace-me"
  }
}

Reload and reinitiate:

sudo swanctl --load-all
sudo swanctl --initiate --child net-net

Trade-off: broad proposals improve interoperability but can negotiate weaker ciphers than you intend. Pin the exact suites both sides support.

Verify it worked:

sudo swanctl --list-sas | egrep 'IKE|ESP|rekey|aes256|sha256'

Path MTU / fragmentation issue inside ESP or UDP 4500

Clamp TCP MSS on traffic entering the tunnel, or lower the tunnel interface MTU if you use a route-based VPN/VTI/XFRM interface.

For iptables:

sudo iptables -t mangle -A FORWARD -o eth0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360
sudo iptables -t mangle -A FORWARD -i eth0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360

Or safer, derive from PMTU:

sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

For nftables:

sudo nft add table inet mangle
sudo nft 'add chain inet mangle forward { type filter hook forward priority mangle; }'
sudo nft add rule inet mangle forward tcp flags syn tcp option maxseg size set rt mtu

For a route-based tunnel interface:

sudo ip link set dev vti0 mtu 1436
ip link show vti0

Edge case: ICMP "Fragmentation Needed" may be filtered, creating PMTU black holes. MSS clamping usually fixes TCP, but UDP apps may still need a lower interface MTU.

Verify it worked:

ping -M do -s 1372 <remote-private-ip> && curl --connect-timeout 5 http://<remote-private-ip>:<port>

Firewall/NAT blocking UDP 500/4500 or ESP, especially after idle timeout

Open the required protocols on both sides and any intermediate firewall. If NAT is present, UDP 4500 must pass; raw ESP may not.

Linux firewall examples:

sudo iptables -A INPUT -p udp --dport 500 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 4500 -j ACCEPT
sudo iptables -A INPUT -p esp -j ACCEPT
sudo iptables -A OUTPUT -p udp --sport 500 -j ACCEPT
sudo iptables -A OUTPUT -p udp --sport 4500 -j ACCEPT
sudo iptables -A OUTPUT -p esp -j ACCEPT

If the peer is behind NAT, prefer NAT-T and do not rely on native ESP through stateful devices with short idle timers. Increase UDP session timeout in your provider's dashboard or firewall policy if that control exists; vendor names vary.

Verify it worked:

sudo tcpdump -ni any 'host <peer-public-ip> and (udp port 500 or udp port 4500 or esp)' -c 20

You should see request/response pairs, not just one-way retransmits.

Traffic selector / subnet mismatch

Correct the local and remote protected subnets on both peers. A common failure is one side using /24 while the other expects /16, or missing a newly added subnet.

strongSwan example:

children {
  net-net {
    local_ts = 10.10.0.0/16,10.11.0.0/16
    remote_ts = 10.20.0.0/16
  }
}

Reload and reinitiate:

sudo swanctl --load-conns
sudo swanctl --terminate --ike net-net
sudo swanctl --initiate --child net-net

If you use policy-based VPN on the other side, update its encryption domain to match exactly.

Verify it worked:

sudo ip xfrm policy | egrep '10.10.|10.11.|10.20.'

DPD/keepalive too aggressive or asymmetric

Relax DPD timers so transient packet loss or short NAT stalls do not tear down the tunnel.

connections {
  net-net {
    dpd_delay = 30s
    dpd_timeout = 120s
    rekey_time = 50m
    over_time = 10m
  }
}

Reload:

sudo swanctl --load-all

Trade-off: longer DPD means slower failover detection. For active/standby designs, keep DPD tight enough for your RTO but not so tight that a few lost keepalives flap production.

Verify it worked:

sudo journalctl -u strongswan --since '30 min ago' --no-pager | egrep 'DPD timeout|liveness check failed|deleting IKE_SA'

No new matches is the goal.

Clock skew causing IKE auth/rekey weirdness

Sync time on both gateways.

sudo timedatectl set-ntp true
chronyc makestep || sudo systemctl restart systemd-timesyncd

If you use certificate auth, re-test after time sync because not-yet-valid or expired cert checks can look like random auth failure.

Verify it worked:

timedatectl status

Look for System clock synchronized: yes.

Prevention

• Add tunnel liveness and rekey monitoring from the gateway, not just app health checks:

*/5 * * * * /usr/sbin/ipsec statusall | /bin/grep -q 'ESTABLISHED' || logger -t vpn 'tunnel down'

• Pin IKE/ESP suites and lifetimes in config management so peers do not drift after one side upgrades defaults:

proposals = aes256-sha256-modp2048
esp_proposals = aes256-sha256-modp2048
life_time = 1h
rekey_time = 50m

• Add a CI lint for subnet selector changes so new private CIDRs cannot ship without VPN policy updates:

grep -R '10\.11\.0\.0/16' infra/ vpn/ || { echo 'missing VPN selector for 10.11.0.0/16'; exit 1; }

• Record PMTU assumptions in firewall config and clamp MSS explicitly on tunnel paths:

sudo iptables-save | grep TCPMSS

• Alert on one-way IKE traffic and retransmits using packet counters or logs. Even a simple journal rule catches this early:

journalctl -u strongswan -f | egrep 'retransmit|NO_PROPOSAL|INVALID_SELECTORS|DPD timeout'

• Keep NTP healthy on gateways and alert when unsynced:

timedatectl show -p NTPSynchronized --value

This article was written by an AI system and published pending human review. Verify anything you intend to act on.