Security Groups vs NACLs vs Firewall Policies: Put Rules in the Right Layer

TL;DR — Most teams should put allow-rules close to the workload with security groups or host firewalls, use network ACLs sparingly for subnet-wide coarse denies, and reserve edge firewall policies for internet-facing filtering and rate/geo controls. The single most common fix is to stop duplicating the same rule in all three layers: pick one primary enforcement point per traffic class, then verify stateful vs stateless behavior before blaming the app. Reading time: ~7 min

What it is and where it sits

You are usually dealing with three different enforcement layers that happen at different points in the request path:

• Security groups: attached to instances, ENIs, NICs, pods, or similar workload-facing interfaces. Usually stateful. Best for "who may talk to this workload on which ports".
• Network ACLs: attached to a subnet/VPC segment boundary. Usually stateless and evaluated in rule order. Best for broad subnet-level guardrails, not app-specific policy.
• Firewall policies: a broad bucket, but in practice this means host firewalls (nftables, iptables, Windows Defender Firewall), edge firewalls (cloud perimeter/WAF/LB policy), or central network firewalls. These sit either on the host, at ingress/egress, or on a transit path.

What they replace depends on your architecture:

• Security groups often replace a pile of hand-maintained host firewall rules for simple service-to-service allow lists.
• NACLs replace router ACL-style segmentation at the subnet boundary.
• Firewall policies replace or augment perimeter appliances, host packet filters, and sometimes L7 controls.

In a typical request flow, they are not interchangeable because they see traffic at different places.

[Internet] ---- 1. TCP 443 ----> [Edge Firewall/LB]
                                   |
                                   | 2. forward to subnet
                                   v
                              [Subnet NACL]
                                   |
                                   | 3. packet to ENI/NIC
                                   v
                            [Security Group]
                                   |
                                   | 4. packet to host
                                   v
                            [Host Firewall]
                                   |
                                   | 5. accept to process
                                   v
                                [App]

If traffic fails, the symptom depends on where it was dropped:

• Edge firewall/LB drop: client may see timeout, TCP reset, 403, or vendor-specific block page.
• NACL drop: usually a silent timeout because stateless packet filtering just discards.
• Security group drop: often timeout; return traffic may work or fail depending on state tracking.
• Host firewall drop/reject: timeout for DROP, immediate Connection refused or ICMP reject for REJECT, depending on rule.

The architecture rule of thumb: put the narrowest, most workload-specific allow rules nearest the workload; put broad, low-churn deny guardrails at network boundaries.

How it actually works

Walk one realistic example: a public API on 443 talks to Postgres on 5432 in a private subnet. The API is reachable from the internet through a load balancer. Developers can SSH only through a bastion or SSM-equivalent, not directly.

Step-by-step packet path

1. A client sends TCP SYN to the public endpoint on 443.
2. The edge firewall/load balancer accepts 443 from the internet and forwards to the API target.
3. The API subnet NACL must allow the inbound ephemeral/load-balanced traffic pattern and the outbound return traffic. Because NACLs are stateless, both directions matter.
4. The API instance security group allows inbound 443 from the load balancer security group or subnet CIDR, and allows outbound 5432 to the DB security group.
5. The host firewall on the API node, if enabled, must also allow 443 to the local process and 5432 egress if you filter output.
6. The API opens a DB connection to 10.0.2.15:5432.
7. The DB subnet NACL allows inbound 5432 from the API subnet and outbound ephemeral return traffic.
8. The DB security group allows inbound 5432 from the API security group, not from the whole subnet if your platform supports SG-to-SG references.
9. The Postgres host firewall allows 5432 from the API subnet or API hosts.
10. Return packets succeed because the SG is stateful, but the NACL still needs explicit return-path rules.

Why people misplace rules

The common failure is opening 5432 in the DB security group and forgetting the DB subnet NACL return path. The app then hangs on connect:

$ psql "host=10.0.2.15 port=5432 dbname=app user=app connect_timeout=3"
psql: error: connection to server at "10.0.2.15", port 5432 failed: timeout expired

From the API host, nc shows the same shape:

$ nc -vz -w 2 10.0.2.15 5432
nc: connect to 10.0.2.15 port 5432 (tcp) timed out: Operation now in progress

If the host firewall is the blocker and uses reject semantics, you get a different symptom:

$ nc -vz -w 2 10.0.2.15 5432
nc: connect to 10.0.2.15 port 5432 (tcp) failed: Connection refused

That difference matters. Timeout usually means packet drop somewhere on path; immediate refusal usually means the host or service actively rejected or nothing is listening.

Concrete diagnosis order

Use this order so you do not bounce randomly between layers:

1. Confirm listener exists on the destination host.

sudo ss -ltnp | grep ':5432'
LISTEN 0      244        0.0.0.0:5432      0.0.0.0:*    users:(("postgres",pid=1873,fd=7))

1. Check host firewall.

sudo nft list ruleset
sudo iptables -S
sudo ufw status verbose

1. Test from source host with a short timeout.

nc -vz -w 2 10.0.2.15 5432
curl -I --connect-timeout 2 https://api.example.com/

1. Inspect security group rules: source, destination, port, protocol, and whether the rule references a CIDR or another SG.
2. Inspect subnet NACLs in both directions. For stateless ACLs, verify ephemeral port ranges on return traffic. On Linux clients that is commonly within 32768-60999 or 32768-65535, but check your OS and platform defaults instead of guessing.
3. Use flow logs or equivalent if your provider has them. They tell you whether the subnet/interface accepted or rejected the packet before it hit the app.

When to use it (and when not to)

Use one primary layer for each job. Do not spread the same intent across all layers unless you need defense in depth for a regulated environment.

You probably do not need a NACL if your goal is "only these two services can talk". That is almost always cleaner as SG-to-SG or host-level policy.

You probably do not need host firewall rules for every app if your platform already gives strong SG semantics and your hosts are ephemeral. A tiny baseline on the host is enough: allow established connections, management channel, and local-only services.

Trade-offs

Every layer buys isolation and costs operability.

• Security groups

  • Benefit: workload-centric, usually stateful, easier to reason about return traffic.
  • Cost: provider-specific semantics, rule sprawl, dependency on tags/ENIs/attachments being correct.

• Network ACLs

  • Benefit: subnet-wide blast-radius control, good for broad deny rules and emergency segmentation.
  • Cost: statelessness, ordered rules, ephemeral port headaches, more false positives during debugging.

• Host firewall policies

  • Benefit: works everywhere, survives provider changes, can protect loopback/local traffic and process-adjacent paths.
  • Cost: config drift, image bake/bootstrapping complexity, harder fleet-wide visibility unless centrally managed.

• Edge firewall policies

  • Benefit: stop junk before it reaches workloads, useful for IP reputation, geo, rate limits, TLS termination adjacency.
  • Cost: another control plane, possible added latency, can hide source IP details if proxying is misconfigured.

Defense in depth is real, but duplicated rules create policy skew: one layer says yes, another says no, and your team loses hours proving which one won.

In practice

Example 1: Linux host firewall with nftables

This accepts established traffic, allows HTTPS to the app, and allows Postgres only from the API subnet.

⚠️ Applying firewall rules over SSH can lock you out immediately. Open a second session first, add an explicit SSH allow rule for your management path, and use a timed rollback mechanism if your distro provides one.

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    iif "lo" accept
    ct state established,related accept
    tcp dport 22 ip saddr 10.0.10.0/24 accept
    tcp dport 443 accept
    tcp dport 5432 ip saddr 10.0.1.0/24 accept
  }

  chain output {
    type filter hook output priority 0; policy accept;
  }
}

Load it with:

sudo nft -f /etc/nftables.conf
sudo nft list ruleset

Gotcha: if your app sits behind a reverse proxy on the same host, you may not need 443 on the app host at all; the proxy may be the only listener.

Example 2: iptables quick test rule for incident isolation

This drops all inbound traffic except established sessions and SSH from a bastion subnet. Useful when you need to quarantine one host fast.

⚠️ This can cause immediate downtime for the isolated service. Run it only on the intended host and record the rollback command before applying it.

sudo iptables -P INPUT DROP
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -p tcp -s 10.0.10.0/24 --dport 22 -j ACCEPT
sudo iptables -S

Rollback:

sudo iptables -F
sudo iptables -P INPUT ACCEPT

Gotcha: this does nothing for cloud load balancer or subnet-level drops. If clients still time out after rollback, the block is elsewhere.

Example 3: Diagnose whether HTTP is blocked at edge vs host

Use curl -I to distinguish a live HTTP responder from a network drop.

$ curl -I --connect-timeout 2 https://api.example.com/
HTTP/2 301 
location: https://www.example.com/
server: nginx

That means network path and TLS are working; your problem is probably app routing or redirect config, not SG/NACL.

Blocked path usually looks like:

$ curl -I --connect-timeout 2 https://api.example.com/
curl: (28) Failed to connect to api.example.com port 443 after 2001 ms: Timeout was reached

Immediate reject often looks like:

$ curl -I --connect-timeout 2 https://api.example.com/
curl: (7) Failed to connect to api.example.com port 443: Connection refused

Gotcha: some edge firewalls intentionally return 403 instead of dropping. A 403 means the path is alive; read the response headers before touching SGs.

Further reading

• Linux nftables wiki and the nft man pages
• Netfilter Packet Filtering HOWTO
• PostgreSQL documentation: "Client Authentication" and "Server Setup and Operation"
• Kubernetes documentation: "Network Policies"
• NIST SP 800-41 Rev. 1, "Guidelines on Firewalls and Firewall Policy"

This article was written by an AI system and published pending human review. Verify anything you intend to act on.