Find Which Firewall Rule Blocked Traffic by Reading Flow Logs

TL;DR — When traffic is blocked and nobody knows which rule did it, start with flow logs closest to the drop point: host firewall logs, then subnet/VPC flow logs, then edge firewall/security policy logs. The most common outcome is a deny on the destination host or subnet from an unexpected source IP/CIDR, so confirm the 5-tuple first and then match it against the logged action and rule ID/name. Reading time: ~6 min

The scenario

It’s Tuesday at 14:37. You deploy a harmless-looking change that moves a worker onto a different subnet, and suddenly the app can’t reach Postgres on 5432. The app logs just say connection timeout, the database graphs look normal, and three people are already saying "security must have changed something" without naming what. You need to answer two questions quickly: where is the packet being dropped, and which rule did it.

Symptoms

• Client-side timeouts instead of explicit rejects:

$ nc -vz db.internal 5432
nc: connect to db.internal (10.42.7.19) port 5432 (tcp) timed out: Operation now in progress

• curl hangs and exits 28 when TCP never completes:

$ curl -sv --connect-timeout 5 http://10.42.7.19:8080/health
*   Trying 10.42.7.19:8080...
* ipv4 connect timeout after 5000ms, move on!
* Failed to connect to 10.42.7.19 port 8080 after 5002 ms: Timeout was reached
* Closing connection 0
curl: (28) Failed to connect to 10.42.7.19 port 8080 after 5002 ms: Timeout was reached

• Application logs show connect timeout, not connection refused:

dial tcp 10.42.7.19:5432: i/o timeout

• No SYN-ACK on packet capture from source host:

$ sudo tcpdump -ni eth0 host 10.42.7.19 and port 5432
14:39:12.101001 IP 10.42.3.44.51182 > 10.42.7.19.5432: Flags [S], seq 238412, win 64240, options [mss 1460,sackOK,TS val 123 ecr 0,nop,wscale 7], length 0
14:39:13.102114 IP 10.42.3.44.51182 > 10.42.7.19.5432: Flags [S], seq 238412, win 64240, options [mss 1460,sackOK,TS val 124 ecr 0,nop,wscale 7], length 0

• Host firewall logs show drops/rejects:

kernel: [UFW BLOCK] IN=eth0 OUT= MAC=... SRC=10.42.3.44 DST=10.42.7.19 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=41721 DF PROTO=TCP SPT=51182 DPT=5432 WINDOW=64240 RES=0x00 SYN URGP=0

• Network flow logs show REJECT/DENY for the 5-tuple:

2026-10-30T14:39:12Z src=10.42.3.44 dst=10.42.7.19 sport=51182 dport=5432 proto=tcp action=deny packets=3 bytes=180 rule=db-deny-app-subnet

• User-facing symptom is usually generic: blank page, gateway timeout, login spinner, API request pending until frontend shows "Network Error".

Likely causes

Step-by-step diagnosis

[Client Host] ---- 1. SYN ----> [Network Policy]
     |                              |
     | 2. tcpdump                   | 3. flow log action/rule
     v                              v
[Source Capture]                [Deny Decision]
     |                              |
     | 4. no reply?                 | 5. match host firewall
     v                              v
[Destination Host] ---- 6. logs ----> [Exact Rule]

1. Confirm the exact 5-tuple from the failing client.

nc -vz db.internal 5432
getent ahostsv4 db.internal
ip route get 10.42.7.19

What means "this is your problem": the resolved IP is not the one you expected, or the route leaves via a different interface/VPN than normal. If DNS is wrong, jump to Fixes → Traffic hit the wrong destination IP because DNS changed. Otherwise keep the exact source IP, destination IP, destination port, and protocol for log searches.

1. Check whether the source host is actually sending SYNs.

sudo tcpdump -ni any 'host 10.42.7.19 and port 5432'

What means "this is your problem": no packets leave at all when you retry the connection. That is not a network policy issue; it is local routing, local firewall, or application binding. If SYNs leave and no SYN-ACK returns, continue.

1. Check destination host firewall logs first; they are the cheapest high-signal source.

sudo journalctl -k --since -15m | egrep 'DROP|REJECT|UFW BLOCK|nft|iptables'
sudo nft list ruleset
sudo iptables -S
sudo ufw status numbered

What means "this is your problem": a log line matching your source IP / destination port, or a default drop policy with no matching allow. Jump to Fixes → Destination host firewall dropped it or Fixes → Rule order / first-match policy shadowed the intended allow.

1. If the host is clean, search subnet/VPC flow logs for the same 5-tuple.

grep 'src=10.42.3.44 dst=10.42.7.19 .*dport=5432 .*proto=tcp' /var/log/flow.log | tail -20

What means "this is your problem": entries with action=deny, REJECT, or similar, especially if they include a rule name/ID. Jump to Fixes → Subnet/VPC/network ACL denied the flow or Fixes → Security group / stateful firewall rule missing source CIDR or port depending on your platform’s control type.

1. Check whether packets arrive at the destination host but replies never leave.

sudo tcpdump -ni any 'host 10.42.3.44 and port 5432'

What means "this is your problem": you see inbound SYNs on the destination, maybe even SYN-ACK leaving, but the client never receives them. That points to return-path filtering, asymmetric routing, or egress deny. Jump to Fixes → Return path blocked.

1. Verify rule order if an allow exists but traffic still drops.

sudo nft list ruleset | sed -n '/table inet/,/}/p'

What means "this is your problem": an earlier broad deny matches before the intended allow, for example a drop ip saddr 10.42.0.0/16 before accept ip saddr 10.42.3.0/24 tcp dport 5432. Jump to Fixes → Rule order / first-match policy shadowed the intended allow.

1. If cloud/provider logs include a rule ID/name, map it back before changing anything. In your provider’s dashboard, open the firewall/security policy page and search for the rule ID/name from the flow log. If the UI varies, use the CLI/API for your provider and filter by rule identifier. What means "this is your problem": the logged rule clearly matches your 5-tuple and action is deny. Fix that rule, not a neighboring one with a similar description.

Fixes

Destination host firewall dropped it (nftables/iptables/ufw/Windows Firewall)

For ufw:

sudo ufw allow from 10.42.3.0/24 to any port 5432 proto tcp
sudo ufw reload
sudo ufw status numbered

For nftables:

sudo nft add rule inet filter input ip saddr 10.42.3.0/24 tcp dport 5432 ct state new,established accept
sudo nft list ruleset > /etc/nftables.conf

For legacy iptables:

sudo iptables -I INPUT 1 -p tcp -s 10.42.3.0/24 --dport 5432 -j ACCEPT
sudo sh -c 'iptables-save > /etc/iptables/rules.v4'

Verify it worked:

nc -vz 10.42.7.19 5432

Subnet/VPC/network ACL denied the flow

Edit the deny/allow list in your provider’s dashboard or CLI so the subnet containing the source can reach the destination subnet/port. Network ACLs are often stateless, so add both directions.

Example rule intent:

{
  "inbound_allow": {"src_cidr": "10.42.3.0/24", "dst_port": 5432, "protocol": "tcp"},
  "outbound_allow": {"dst_cidr": "10.42.3.0/24", "src_port": 5432, "protocol": "tcp"}
}

If your platform uses numbered ACLs, insert the allow before the broad deny.

⚠️ Changing subnet/firewall policy can open access for multiple hosts at once. Scope the CIDR to the smallest real source range and record the change ticket before applying it.

Verify it worked:

grep 'src=10.42.3.44 dst=10.42.7.19 .*dport=5432' /var/log/flow.log | tail -5

You want to see accept/allow, not deny.

Security group / stateful firewall rule missing source CIDR or port

Add the missing inbound allow on the destination workload and, if your platform requires it, the matching egress allow on the source workload.

Typical intent:

{
  "direction": "ingress",
  "protocol": "tcp",
  "port": 5432,
  "source": "10.42.3.0/24",
  "target": "db-servers"
}

Common mistake: the app moved from 10.42.2.0/24 to 10.42.3.0/24, but the rule still only allows the old subnet.

Verify it worked:

sudo tcpdump -ni any 'host 10.42.7.19 and port 5432' -c 4

You want to see S, then S. back, then ..

Traffic hit the wrong destination IP because DNS changed

Check and correct the record in your DNS control plane. In your provider’s dashboard, go to DNS/Records for the zone and update db.internal to the correct A/AAAA record, or fix the service discovery registration.

getent ahostsv4 db.internal
dig +short db.internal A

If clients cache aggressively, flush local cache or wait for TTL expiry.

Verify it worked:

getent ahostsv4 db.internal && nc -vz db.internal 5432

Return path blocked (asymmetric routing / egress deny)

On the destination host, check routing and reverse path filtering.

ip route get 10.42.3.44
sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.eth0.rp_filter

If replies leave via the wrong interface, fix the route or policy routing. If strict reverse path filtering is dropping valid asymmetric traffic, set loose mode only if your network design requires it:

sudo sysctl -w net.ipv4.conf.all.rp_filter=2
sudo sysctl -w net.ipv4.conf.eth0.rp_filter=2

Persist it:

printf 'net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.eth0.rp_filter=2
' | sudo tee /etc/sysctl.d/99-rpf.conf
sudo sysctl --system

Also check egress firewall rules on the destination for replies.

Verify it worked:

sudo tcpdump -ni any 'host 10.42.3.44 and port 5432' -c 6

You want to see the full handshake, not repeated SYNs.

Rule order / first-match policy shadowed the intended allow

Move the specific allow above the broad deny. For nftables:

sudo nft insert rule inet filter input position 1 ip saddr 10.42.3.0/24 tcp dport 5432 accept
sudo nft list chain inet filter input
sudo nft list ruleset > /etc/nftables.conf

For iptables:

sudo iptables -I INPUT 1 -p tcp -s 10.42.3.0/24 --dport 5432 -j ACCEPT
sudo iptables -S INPUT

Verify it worked:

sudo journalctl -k --since -5m | egrep '10.42.3.44|5432'

You should stop seeing matching drop logs for that flow.

Prevention

• Enable flow logs on every segmented boundary and include action plus rule identifier if your platform supports it. Store them centrally and index on src, dst, dport, proto, action, rule.

{"timestamp":"2026-10-30T14:39:12Z","src":"10.42.3.44","dst":"10.42.7.19","sport":51182,"dport":5432,"proto":"tcp","action":"deny","rule":"db-deny-app-subnet"}

• Add a CI check that diffs firewall policy and fails on widened CIDRs or removed app-to-db allows.

git diff --exit-code origin/main -- infra/firewall/ || { echo 'Firewall policy changed'; exit 1; }

• Pin source ranges in inventory/service discovery instead of hand-entered subnets in rules. Generate rules from source-of-truth.

app_subnets:
  - 10.42.3.0/24
  - 10.42.4.0/24
db_port: 5432

• Add synthetic connectivity tests from each trust zone to critical dependencies.

nc -zw3 db.internal 5432 || echo 'db path failed'
curl -fsS --connect-timeout 3 http://api.internal/health || echo 'api path failed'

• Log host firewall drops with a recognizable prefix so grep works during incidents.

sudo iptables -A INPUT -j LOG --log-prefix 'IPTABLES DROP ' --log-level 4

• Document rule ownership in descriptions/comments so a flow-log rule ID maps to a team immediately.

# not nginx config; example metadata convention
rule_name = "db-allow-app-subnet"
owner = "payments-platform"
change_ref = "CHG-4821"

This article was written by an AI system and published pending human review. Verify anything you intend to act on.