Build Kubernetes and VM Microsegmentation from Observed Traffic

TL;DR — Start by observing real traffic, not by guessing dependencies. Export a short window of flows, group them by workload, write allow-only rules for the traffic you actually saw, apply them first in audit/staging, then enforce with Kubernetes NetworkPolicy for pods and nftables/ipset rules for VMs. Reading time: ~5 min

Goal

When you finish, your Kubernetes namespaces and your Linux virtual machines will only accept the east-west traffic you explicitly observed and allowed, and you will be able to prove it with kubectl, curl, nc, and firewall counters showing allowed paths still work and unapproved paths fail.

Prerequisites

• Kubernetes cluster access with permission to read pods and apply NetworkPolicy objects — check with:

kubectl auth can-i get pods --all-namespaces && kubectl auth can-i apply networkpolicies --all-namespaces

• A CNI that enforces NetworkPolicy for ingress and egress. Check your nodes and CNI pods:

kubectl get pods -n kube-system -o wide

• Linux VMs with root or sudo access and nft available — check with:

nft --version

• kubectl >= 1.29 — check with:

kubectl version --client --short

• jq, curl, and nc installed locally:

jq --version && curl --version | head -n1 && nc -h 2>&1 | head -n1

• A recent sample of observed traffic. Any source is fine if it gives you src, dst, port, and protocol: VPC flow logs, node conntrack, eBPF flow export, service mesh telemetry, or app-side access logs.
• A maintenance window or a staging environment. Enforcing deny rules can break production traffic immediately.

Steps

Step 1: Export the flows you will build policy from

Use a simple CSV with one row per observed connection. Create flows.csv with literal columns src,dst,proto,port.

cat > flows.csv <<'CSV'
src,dst,proto,port
web,api,tcp,8080
api,postgres,tcp,5432
web,dns,tcp,53
web,dns,udp,53
api,dns,tcp,53
api,dns,udp,53
CSV

If you already have raw logs, reduce them to unique flows:

awk -F, 'NR==1 || !seen[$1 FS $2 FS $3 FS $4]++' flows.csv > flows-uniq.csv
column -s, -t < flows-uniq.csv

You should see one line per allowed dependency, for example web api tcp 8080.

Step 2: Map flow names to Kubernetes labels and VM IPs

Create one file for Kubernetes label selectors and one for VM addresses.

cat > k8s-map.json <<'JSON'
{
  "web": {"namespace": "prod", "podSelector": {"matchLabels": {"app": "web"}}},
  "api": {"namespace": "prod", "podSelector": {"matchLabels": {"app": "api"}}},
  "postgres": {"namespace": "prod", "podSelector": {"matchLabels": {"app": "postgres"}}}
}
JSON

cat > vm-map.json <<'JSON'
{
  "dns": ["10.0.0.53", "10.0.0.54"],
  "legacy-batch": ["10.0.2.20"]
}
JSON

Confirm your labels exist before writing policy:

kubectl get pods -n prod --show-labels | egrep 'app=(web|api|postgres)'

You should see the target pods with the exact labels used in k8s-map.json.

Step 3: Apply a default deny in one namespace

⚠️ This can cause downtime if you apply it before adding allow rules for DNS, health checks, and required service-to-service traffic.

Apply default deny for both ingress and egress in the target namespace.

cat > default-deny-prod.yaml <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: prod
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
YAML

kubectl apply -f default-deny-prod.yaml

Success looks like:

networkpolicy.networking.k8s.io/default-deny-all created

Step 4: Add Kubernetes allow rules from the observed flows

Apply literal policies for the observed pod-to-pod and pod-to-DNS traffic.

cat > observed-allow.yaml <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-allow-egress-api-and-dns
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes: [Egress]
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 8080
  - to:
    - ipBlock:
        cidr: 10.0.0.53/32
    - ipBlock:
        cidr: 10.0.0.54/32
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-ingress-web-and-egress-db-dns
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: web
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  - to:
    - ipBlock:
        cidr: 10.0.0.53/32
    - ipBlock:
        cidr: 10.0.0.54/32
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-ingress-api
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 5432
YAML

kubectl apply -f observed-allow.yaml

You should see three created or configured lines.

Step 5: Enforce equivalent rules on VMs with nftables

On each VM, create sets for approved source CIDRs and allow only the observed ports. Replace CIDRs with your pod CIDRs or node CIDRs that actually originate the traffic.

sudo nft -f - <<'NFT'
table inet microseg {
  set dns_clients_v4 {
    type ipv4_addr;
    elements = { 10.244.0.0/16 }
  }
  chain input {
    type filter hook input priority 0; policy drop;
    iif "lo" accept
    ct state established,related accept
    ip saddr @dns_clients_v4 udp dport 53 accept
    ip saddr @dns_clients_v4 tcp dport 53 accept
    counter reject with icmpx type admin-prohibited
  }
}
NFT

sudo nft list ruleset | sed -n '/table inet microseg/,$p'

You should see policy drop and the explicit accept rules you just installed.

Step 6: Test allowed and denied paths before broad rollout

Run one allowed test and one denied test from inside the cluster.

kubectl run nettest --rm -it -n prod --image=curlimages/curl:8.8.0 --restart=Never -- sh

Inside the shell, run:

curl -sS --connect-timeout 3 http://api:8080/health
nc -vz -w 3 postgres 5432
nc -vz -w 3 postgres 3306

Expected output shape:

OK
postgres (10.96.142.17:5432) open
nc: postgres (10.96.142.17:3306): Operation timed out

You should see the approved port succeed and the unapproved port fail.

[web pod] ---- 1. TCP 8080 ----> [api pod]
[api pod] ---- 2. TCP 5432 ----> [postgres pod]
[web pod] ---- 3. UDP/TCP 53 --> [dns VM]
[web pod] ---- 4. TCP 3306 ----> [postgres pod]
                           |
                           v
                        [DENY]

Verify it works

List the active policies and verify the namespace is no longer wide open:

kubectl get networkpolicy -n prod
kubectl describe networkpolicy -n prod web-allow-egress-api-and-dns

Expected shape:

NAME                                POD-SELECTOR   AGE
default-deny-all                    <none>         2m
web-allow-egress-api-and-dns        app=web        90s
api-allow-ingress-web-and-egress... app=api        90s
postgres-allow-ingress-api          app=postgres   90s

Verify VM counters increment on allowed traffic:

sudo nft list chain inet microseg input

Expected shape after tests:

chain input {
  type filter hook input priority filter; policy drop;
  iif "lo" accept
  ct state established,related accept
  ip saddr @dns_clients_v4 udp dport 53 accept
  ip saddr @dns_clients_v4 tcp dport 53 accept
  counter packets 12 bytes 984 reject with icmpx type admin-prohibited
}

Run an end-to-end denied-path check from a pod:

kubectl run nettest2 --rm -it -n prod --image=busybox:1.36 --restart=Never -- sh -c 'timeout 3 nc -vz postgres 3306; echo exit=$?'

Expected result:

nc: postgres (10.96.142.17:3306): Operation timed out
exit=1

Common pitfalls

CNI does not enforce egress NetworkPolicy

Mistake: applying NetworkPolicy on a cluster whose CNI only partially enforces policy.

Symptom: kubectl get networkpolicy looks correct, but denied egress still succeeds.

Fix: run a real deny test with nc; if it still connects, switch to a CNI that enforces ingress and egress or enable that feature in your current CNI.

You forgot DNS

Mistake: default deny applied before adding UDP/TCP 53 egress.

Symptom: apps fail with name resolution errors such as:

curl: (6) Could not resolve host: api.internal

Fix: add explicit egress allows for both UDP 53 and TCP 53 to your DNS IPs, then re-test.

Service IPs were allowed instead of pod selectors

Mistake: writing policy against a ClusterIP that changes or is bypassed by direct pod traffic.

Symptom: traffic works one moment and fails after rollout or rescheduling.

Fix: for pod-to-pod traffic, use podSelector and namespaceSelector; reserve ipBlock for real external or VM IPs.

Host firewall drops return traffic

Mistake: VM nftables rules omit ct state established,related accept.

Symptom: the first packet reaches the VM, but clients hang or retry until timeout.

Fix: add this literal rule near the top of the input chain:

sudo nft add rule inet microseg input ct state established,related accept

Flow window was too short

Mistake: generating policy from five minutes of daytime traffic and missing cron jobs, migrations, or backup paths.

Symptom: scheduled jobs fail later with connection timeouts on ports you never allowed.

Fix: collect flows over at least one full business cycle plus scheduled jobs, then merge unique flows before enforcing.

Labels changed after deployment

Mistake: policy selects app=api but the deployment now uses app.kubernetes.io/name=api.

Symptom: policy exists, but it matches zero pods:

kubectl describe networkpolicy -n prod api-allow-ingress-web-and-egress-db-dns

Fix: update the podSelector.matchLabels to the labels shown by:

kubectl get pods -n prod --show-labels

This article was written by an AI system and published pending human review. Verify anything you intend to act on.