WAF rule update blocks legitimate requests: diagnose and fix false positives

TL;DR — A WAF rule update usually breaks legitimate traffic in one of four ways: a managed rule/signature started matching your app, a custom rule is too broad, the WAF no longer trusts your proxy/CDN IPs, or request normalization changed and now trips inspection. The fastest safe fix is to identify the blocking rule ID from WAF logs, disable or narrow that one rule for the affected path/method/content type, then verify with a replayed request. Reading time: ~6 min

The scenario

It is 14:37 on a Tuesday. You merged what looked like a harmless security change: updated managed WAF rules and one custom expression to tighten API filtering. Five minutes later, checkout POSTs start failing, mobile clients get 403s on login, and support says "desktop works if you refresh twice". Your app pods are healthy, database latency is normal, and the only thing that changed in the request path is the WAF policy.

Symptoms

• Users see 403 Forbidden, 406 Not Acceptable, or a branded block page before the app responds.
• API clients get a short edge response body instead of your normal JSON, for example:

HTTP/2 403 
content-type: text/html
server: <edge-or-waf>
content-length: 5621

<html><title>Request blocked</title>...

• curl against the origin succeeds, but the same request through the public hostname fails:

curl -si https://api.example.com/login -X POST -H 'content-type: application/json' --data '{"email":"a@b.co","password":"x"}'

HTTP/2 403 
content-type: text/html
x-request-id: 7f2c...

<html>...blocked by security policy...</html>

curl -si http://origin.internal:8080/login -X POST -H 'content-type: application/json' --data '{"email":"a@b.co","password":"x"}'

HTTP/1.1 200 OK
content-type: application/json

{"ok":true}

• WAF or reverse-proxy logs show rule hits such as:

{"action":"block","rule_id":"942100","message":"SQL Injection Attack Detected via libinjection","path":"/login","method":"POST","status":403}

{"action":"block","rule_id":"custom-allow-api-json","message":"Request denied by custom expression","path":"/v1/orders","method":"POST","status":403}

• Nginx/ingress logs at the app layer show fewer requests than usual because they never reach origin.
• Error rate spikes only on specific methods, paths, countries, content types, or user agents after the rule update timestamp.

Likely causes

Step-by-step diagnosis

1. Confirm the block happens at the edge, not in the app.

curl -si https://app.example.com/affected-path -X POST -H 'content-type: application/json' --data '{"probe":1}' | sed -n '1,20p'

If you see a 403/406 with an HTML block page, a server header from your edge, or a response body that is not your app's normal format, this is a WAF/edge problem. Jump to step 2.

1. Bypass the WAF and hit origin directly from a trusted network.

curl -si http://ORIGIN_IP:8080/affected-path -H 'Host: app.example.com' -X POST -H 'content-type: application/json' --data '{"probe":1}' | sed -n '1,20p'

If origin returns 2xx/4xx in your app's normal shape while the public hostname returns a WAF block, the WAF is the fault domain. Jump to step 3.

1. Pull the exact blocking rule ID/message from logs.

grep -R -Ei 'block|den(y|ied)|rule_id|signature|anomaly' /var/log 2>/dev/null | tail -n 100

This is your problem if you find entries with a rule ID, signature, or category tied to the affected path and timestamp, for example 942100, SQL Injection, bot, or a custom rule name. If the hit is a managed signature, jump to "Managed rule/signature false positive". If it names your custom rule, jump to "Custom WAF rule/expression is too broad".

1. Check whether the source IP seen by the WAF changed after the update.

awk '/X-Forwarded-For|CF-Connecting-IP|True-Client-IP|remote_addr/ {print}' /var/log/nginx/access.log | tail -n 20

This is your problem if all requests suddenly appear from your CDN/load balancer IP range, a private subnet, or the same IP, and the blocked rule is geo/IP/rate based. Jump to "Trusted proxy/CDN IP list is stale".

1. Replay a blocked request exactly, including encoding and body.

curl --trace-ascii /tmp/trace.txt -si 'https://app.example.com/v1/search?q=%2527select%2527' -H 'content-type: application/json' --data '{"query":"name:foo/bar","filters":["a+b","x%2Fy"]}' > /tmp/resp.txt
sed -n '1,80p' /tmp/resp.txt

This is your problem if only certain encodings, JSON fields, GraphQL queries, or multipart uploads trigger the block, especially after a parser/normalization update. Jump to "Request normalization/body inspection changed".

1. Check for rate-limit or bot-policy hits.

grep -R -Ei 'rate limit|bot|threshold|too many requests|429' /var/log 2>/dev/null | tail -n 50

This is your problem if real users share NATed IPs, mobile app traffic lacks a browser fingerprint, or login bursts now get blocked/challenged. Jump to "Rate-limit/bot rule update is catching real users".

[Client] ---- 1. blocked 403 ----> [WAF]
  ^                                |
  | 5. replay exact request        | 3. rule hit logs
  |                                v
[Origin direct curl] <---- 2. bypass test ---- [Origin]

Fixes

Managed rule/signature false positive after ruleset update

Prefer a scoped exclusion over disabling the whole ruleset.

If you run ModSecurity/OWASP CRS with nginx, exclude the specific rule for the affected location:

location /login {
    ModSecurityEnabled on;
    ModSecurityRules '
      SecRuleRemoveById 942100
    ';
    proxy_pass http://app_backend;
}

Or exclude by target instead of whole rule if one JSON field is noisy:

SecRuleUpdateTargetById 942100 "!ARGS:password"

Then reload:

nginx -t && systemctl reload nginx

If your WAF is provider-managed, disable only the named rule/signature for the affected path/method in the dashboard's WAF policy editor, not globally.

Verify it worked:

curl -si https://app.example.com/login -X POST -H 'content-type: application/json' --data '{"email":"a@b.co","password":"x"}' | sed -n '1,15p'

Custom WAF rule/expression is too broad

Find the last changed custom rule and narrow it with path, method, host, and content-type conditions. Bad pattern: block all requests containing select anywhere. Better pattern: scope to suspicious query strings on public GET endpoints.

Example before:

{"if":"lower(uri) contains \"select\" or body contains \"union\"","action":"block"}

Example after:

{"if":"method == \"GET\" and starts_with(path, \"/search\") and lower(query_string) contains \" union \"","action":"block"}

For nginx with ModSecurity, replace a broad custom rule:

SecRule REQUEST_URI|ARGS|REQUEST_BODY "@rx (?i:select|union|sleep\()" "id:100100,phase:2,deny,status:403,msg:'broad sqli block'"

with a scoped one:

SecRule REQUEST_METHOD "@streq GET" "id:100100,phase:2,pass,nolog,chain"
SecRule REQUEST_URI "@beginsWith /search" "chain"
SecRule QUERY_STRING "@rx (?i:(?:\bunion\b.{0,20}\bselect\b|sleep\())" "deny,status:403,msg:'scoped sqli block'"

Verify it worked:

curl -si 'https://app.example.com/v1/orders' -X POST -H 'content-type: application/json' --data '{"selection":"standard"}' | sed -n '1,15p'

Trusted proxy/CDN IP list is stale, so client IP/headers look wrong to the WAF

If the WAF or origin stopped trusting your edge proxy, all requests may appear to come from the proxy IP, breaking geo/IP allowlists and rate limits.

For nginx, update trusted proxies and real IP header:

set_real_ip_from 203.0.113.0/24;
set_real_ip_from 198.51.100.0/24;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Test and reload:

nginx -t && systemctl reload nginx

If your provider publishes edge IP ranges, sync them automatically instead of hard-coding stale values. Also verify your load balancer forwards the expected header (X-Forwarded-For, True-Client-IP, or provider equivalent) and your WAF policy keys on the same header.

Verify it worked:

tail -n 5 /var/log/nginx/access.log

You should see real client IPs again, not only proxy ranges.

Request normalization/body inspection changed and now flags valid JSON, GraphQL, multipart, or encoded URLs

The safe fix is to reduce inspection scope for the affected endpoint or content type, not to disable body inspection globally.

For a GraphQL endpoint behind ModSecurity, exclude request body inspection on that path and keep header/path checks:

location /graphql {
    ModSecurityEnabled on;
    ModSecurityRules '
      SecRuleRemoveById 942100 942110 941100
    ';
    proxy_pass http://app_backend;
}

For file uploads, raise body limits if the update lowered them:

SecRequestBodyAccess On
SecRequestBodyLimit 134217728
SecRequestBodyNoFilesLimit 1048576

If double-encoded URLs are legitimate in your app, normalize them once at the app/router and stop sending ambiguous encodings from clients. Do not allow mixed encodings on security-sensitive endpoints unless you have tests for them.

Verify it worked:

curl -si 'https://app.example.com/graphql' -H 'content-type: application/json' --data '{"query":"query { viewer { id } }"}' | sed -n '1,15p'

Rate-limit/bot rule update is catching real users or mobile clients

Raise thresholds only for the affected route/population, or key on a better identifier than source IP.

If your reverse proxy enforces rate limits, scope them by path and use a header/API key when available:

limit_req_zone $binary_remote_addr zone=login_ip:10m rate=5r/s;
limit_req_zone $http_authorization zone=api_token:10m rate=20r/s;

location /login {
    limit_req zone=login_ip burst=20 nodelay;
}

location /api/ {
    limit_req zone=api_token burst=50 nodelay;
}

If mobile clients are being classified as bots, exempt the app's authenticated API paths from browser-style bot checks and keep abuse controls at auth, token, and per-account layers.

Verify it worked:

for i in $(seq 1 10); do curl -s -o /dev/null -w '%{http_code}\n' https://app.example.com/login; done

You should stop seeing immediate 403/429 for normal burst patterns.

Prevention

• Pin WAF ruleset versions and roll them out like app deploys. Store the version in config and promote dev -> staging -> prod explicitly.

waf:
  managed_ruleset: "crs-4.10"
  custom_policy_sha256: "9f2d..."

• Add a canary replay job in CI/CD that sends known-good requests through the public edge after any WAF change.

curl -fsS https://app.example.com/healthz >/dev/null
curl -fsS https://app.example.com/login -X POST -H 'content-type: application/json' --data @ci/fixtures/login.json >/dev/null
curl -fsS https://app.example.com/graphql -H 'content-type: application/json' --data @ci/fixtures/graphql.json >/dev/null

Fail the deploy on non-2xx.

• Alert on WAF blocks by rule ID and path, not just total 4xx. Example log query dimensions: rule_id, host, path, method, content_type, country, client_ip_class.

{"alert":"waf_false_positive_suspect","condition":"rate(blocks{path='/login',rule_id='942100'}[5m]) > 20"}

• Keep trusted proxy IP ranges synced automatically and test them daily.

./sync-edge-ips.sh && nginx -t && systemctl reload nginx

Also run a smoke check that compares remote_addr vs X-Forwarded-For handling in logs.

• Add endpoint-specific exclusions as code with comments and expiry review dates.

# TEMP until GraphQL parser upgrade lands, review 2026-12-01
SecRuleRemoveById 942100 "phase:2,id:100901,ctl:ruleRemoveById=942100"

• Capture one blocked sample request safely for replay. Log request metadata and a hashed body fingerprint, not raw secrets.

{"path":"/login","method":"POST","content_type":"application/json","body_sha256":"...","rule_id":"942100","action":"block"}

This article was written by an AI system and published pending human review. Verify anything you intend to act on.