Feature flags done right: rollout, kill switches, and flag removal

TL;DR — Use feature flags to decouple deploy from release, but treat every flag as temporary operational code with an owner, expiry, and removal plan. The most likely fix for teams drowning in flags is to classify flags up front (release vs kill switch vs experiment), keep evaluation local and fast, and create the removal ticket the same day you create the flag. Reading time: ~7 min

What it is and where it sits

A feature flag is not just an if statement in app code. In production, it becomes part of your control plane: product or ops changes flag state, your app evaluates that state during request handling or job execution, and metrics tell you whether to continue, pause, or roll back.

In a typical system, the flag service or config store sits beside your application config, not in the hot path of every request. Good implementations fetch flag definitions out of band, cache them in memory, and evaluate locally. What it replaces is the old pattern of “deploy code, hope it works, redeploy to disable it.” For risky changes, the flag becomes a faster, narrower rollback mechanism than shipping a revert commit.

Typical request/data flow:

[operator/product] --> [flag config store/control plane]
                              |
                              v
                    [SDK/cache in app process]
                              |
client --> [load balancer] --> [app request handler] --> [flag eval]
                                                      | true
                                                      v
                                                [new code path]
                                                      |
                                                      v
                                                  [DB/API]
                                                      ^
                                                      | false
                                                [old code path]

What talks to it:

• Your web app / API handlers
• Background workers and cron jobs
• Sometimes frontend clients, but only for non-sensitive UI behavior
• Observability pipeline: metrics, logs, traces tagged with flag state

Where it lives:

• Flag definitions: config store, database table, or vendor-hosted control plane
• Evaluation: usually in-process in the app or worker
• Audit trail: your change-management path, app logs, or provider audit logs

What experienced teams avoid: making a remote HTTP call to a flag service on every request. That turns a release safety mechanism into a latency and outage dependency.

How it actually works

Walk one realistic example: rolling out a new checkout tax calculation service to 5%, then 25%, then 100%, with a kill switch.

Assume:

• Existing path: app calculates tax internally
• New path: app calls tax-service
• Risk: wrong totals, elevated latency, third-party failures
• Goal: enable for a subset of users, instantly disable if errors spike

Step 1: Define the flag and its metadata

Create a flag with explicit metadata, not just a name. Minimum useful fields:

• key: checkout.tax_service_v2
• type: release
• owner: team or person
• created_at
• expires_at
• default: false
• rollout rule: percentage by stable user key
• kill-switch semantics: force old path when off

The important design choice is stable bucketing. If 5% of users get the new path today, the same users should keep getting it tomorrow unless you change the percentage. Hash a stable identifier like account_id or user_id; do not use random per-request assignment.

Step 2: App fetches config out of band and caches it

At startup, the app loads flag config from a local file, database, or provider SDK and refreshes periodically in a background thread. Request handlers read from in-memory state.

If the control plane is unreachable, your app should keep serving using the last known config or a safe default. For a risky release flag, “safe default” usually means old behavior.

Step 3: Evaluate locally inside the request

Request comes in for POST /checkout. Handler evaluates checkout.tax_service_v2 for user_id=12345.

Pseudo-flow:

1. Read current flag definition from memory.
2. If a global override says OFF, use old path immediately.
3. Else compute bucket from user_id hash.
4. If bucket < rollout percentage, use new tax service.
5. Emit metrics with flag=checkout.tax_service_v2 and variant=on|off.

The kill switch is just an override with higher precedence than percentage rollout. That precedence rule matters during incidents.

Step 4: Observe before increasing rollout

You should compare at least:

• error rate
• p95/p99 latency
• business correctness metrics if available, like tax mismatch or payment authorization drop
• downstream dependency health

If you use logs, include the flag decision explicitly. Example log shape:

{"ts":"2026-08-10T14:22:31Z","route":"POST /checkout","user_id":12345,"flag":"checkout.tax_service_v2","variant":"on","tax_ms":183,"status":200}

If the new path starts failing, your first action is not a redeploy. Flip the kill switch to OFF.

Step 5: Kill switch during an incident

Suppose tax-service starts timing out. Your app should fail closed to the old path if the flag is disabled, and ideally also have a code-level timeout/circuit breaker so “flag on” does not mean “hang forever.”

A bad operational pattern is a flag that only hides the UI while workers or APIs still execute the new path. A real kill switch disables all entry points: HTTP handlers, workers, scheduled jobs, and async consumers.

Step 6: Finish rollout and delete the flag

Once the rollout is at 100% and stable for an agreed window, stop treating the flag as operationally useful. Remove it.

The sequence that avoids surprises:

1. Set rollout to 100% for a few days.
2. Delete targeting rules and overrides; leave a constant true in non-prod if needed for one release.
3. Remove old code path.
4. Remove the flag evaluation call.
5. Delete the flag definition from the control plane/config store.
6. Remove dashboards/alerts that reference the flag.

If you skip step 5, someone will eventually flip a dead flag and think they changed behavior when they did not.

When to use it (and when not to)

You probably do not need feature flags if the change is low-risk, rollback is cheap, and the flag would outlive the feature. Every flag adds branching, test matrix growth, and operational ambiguity.

Trade-offs

Benefit and cost come together here:

• Faster release control without redeploying
  • Cost: more runtime branches, more states to test, more incident playbooks to maintain
• Percentage rollout reduces blast radius
  • Cost: harder debugging because behavior differs by user/account; support needs visibility into flag state
• Kill switch can stop a bad dependency quickly
  • Cost: only works if every execution path honors it, including workers and retries
• Decouples deploy from release for product teams
  • Cost: operational control plane, audit requirements, and possible vendor lock-in if you use a hosted system
• Local evaluation keeps latency low
  • Cost: eventual consistency; a flag flip may take seconds to propagate unless you use streaming/push
• Rich targeting rules
  • Cost: complexity and surprise. “User in region X on plan Y but not cohort Z” becomes impossible to reason about during incidents

Latency and availability specifics:

• In-process evaluation: microseconds to low milliseconds, usually fine
• Remote per-request evaluation: adds network latency and introduces a new failure mode; avoid it for hot paths
• Cache staleness: if refresh interval is 30s, your kill switch is not truly instant unless you have push updates

Lock-in concern:

• If flag rules are embedded deeply in a vendor SDK format, migration gets painful
• Keep your app-side interface small: isEnabled(flagKey, context) and your own metadata conventions

In practice

Example 1: Express/Node.js local evaluation with percentage rollout and kill switch

import crypto from "node:crypto";

const flags = {
  "checkout.tax_service_v2": {
    enabled: true,
    killSwitch: false,
    rolloutPercent: 5,
    salt: "checkout-tax-v2"
  }
};

function bucketUser(userId, salt) {
  const hex = crypto.createHash("sha256").update(`${salt}:${userId}`).digest("hex");
  const n = parseInt(hex.slice(0, 8), 16);
  return n % 100;
}

export function isEnabled(flagKey, ctx) {
  const flag = flags[flagKey];
  if (!flag || !flag.enabled) return false;
  if (flag.killSwitch) return false;
  if (!ctx.userId) return false;
  return bucketUser(ctx.userId, flag.salt) < flag.rolloutPercent;
}

export async function checkoutHandler(req, res) {
  const on = isEnabled("checkout.tax_service_v2", { userId: req.user.id });
  req.log.info({ flag: "checkout.tax_service_v2", variant: on ? "on" : "off" });

  const tax = on
    ? await calculateTaxViaService(req.body).catch(() => calculateTaxLegacy(req.body))
    : await calculateTaxLegacy(req.body);

  res.json({ ok: true, tax });
}

This does local, stable percentage rollout and a hard kill switch. Gotcha: if you change the salt or the identity key, your cohorts reshuffle and your 5% is no longer the same 5%.

Example 2: Postgres-backed flag table with explicit expiry metadata

⚠️ Editing production flags directly in the database can change live behavior immediately. Run the UPDATE in a transaction, target a single key, and verify the row count before commit.

CREATE TABLE feature_flags (
  key text PRIMARY KEY,
  enabled boolean NOT NULL DEFAULT false,
  kill_switch boolean NOT NULL DEFAULT false,
  rollout_percent integer NOT NULL DEFAULT 0 CHECK (rollout_percent BETWEEN 0 AND 100),
  owner text NOT NULL,
  expires_at timestamptz NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now()
);

INSERT INTO feature_flags (key, enabled, kill_switch, rollout_percent, owner, expires_at)
VALUES ('checkout.tax_service_v2', true, false, 5, 'payments-team', now() + interval '14 days');

BEGIN;
UPDATE feature_flags
SET kill_switch = true, updated_at = now()
WHERE key = 'checkout.tax_service_v2';
SELECT key, enabled, kill_switch, rollout_percent, updated_at
FROM feature_flags
WHERE key = 'checkout.tax_service_v2';
COMMIT;

This gives you a minimal control plane with owner and expiry. Gotcha: if your app polls every 60 seconds, kill_switch=true is not immediate; lower the poll interval or use notifications for urgent flags.

Example 3: Removal checklist enforced in CI

#!/usr/bin/env bash
set -euo pipefail

expired_flags=$(psql "$DATABASE_URL" -Atc "SELECT key FROM feature_flags WHERE expires_at < now()")

if [[ -n "$expired_flags" ]]; then
  echo "Expired feature flags found:"
  echo "$expired_flags"
  exit 1
fi

echo "No expired feature flags."

Typical failure output shape:

Expired feature flags found:
checkout.tax_service_v2
legacy.invoice_pdf_path

This forces cleanup pressure into CI instead of relying on memory. Gotcha: do not block emergency hotfix pipelines on unrelated expired flags unless your process can handle that friction; many teams run this on main branch or as a scheduled job that opens tickets.

Further reading

• Martin Fowler, "Feature Toggles"
• Google SRE Book, the chapters on "Release Engineering" and "Handling Overload"
• MDN Web Docs, the "HTTP Caching" chapter
• PostgreSQL Documentation, "LISTEN/NOTIFY"
• OpenTelemetry Documentation, semantic conventions for logs, metrics, and traces

This article was written by an AI system and published pending human review. Verify anything you intend to act on.