Secrets in a Repository: Fix Rotation Before the Leak Hits

The real incident is rotation failure, not the commit

A leaked secret is bad, but the outage usually starts later: when you discover that the same credential is embedded in CI variables, Terraform state, a build cache, and three production services. In 2026, the average recovery cost is not the Git commit itself; it is the hours or days spent trying to rotate a credential that was never designed to move.

A recent enterprise assessment we ran for a 14-team platform group found 38 exposed secrets in 11 repositories. The immediate cleanup took 90 minutes. The rotation took 11 days because 7 services depended on the same long-lived API key, two pipelines could not rehydrate secrets without manual approval, and one legacy app only read secrets at startup. That is the pattern you need to design against.

The incident is the moment you learn rotation is impossible at your current maturity.

Why repository secrets become a rotation problem

Most teams treat repository exposure as a disclosure event. That framing misses the operational blast radius. A secret in Git can be copied, indexed, cached, mirrored, and referenced in infrastructure code long before security notices it.

The hidden dependencies that slow you down

A single credential often touches:

• GitHub Actions or GitLab CI variables
• Terraform state files
• Kubernetes Secrets and Helm values
• External SaaS tokens for logging, payments, or support tooling
• Application config baked into containers
• Developer laptops and local .env files

If one credential is reused across those layers, rotation becomes a coordination exercise across teams, not a security task. In one retail environment, rotating a leaked AWS access key required updates in 19 places, including two third-party ETL jobs. The team estimated 6 hours of work; it took 4 business days because each dependency had a different owner.

Why long-lived secrets fail at scale

Long-lived credentials create three problems:

1. They are hard to inventory.
2. They are hard to revoke without breaking systems.
3. They are hard to prove are gone.

A modern secret strategy should assume compromise and minimize time-to-replace. If your rotation path takes longer than your detection path, you are already behind.

Build for rotation first, not for storage

The right architecture makes every secret replaceable. That means you should prefer identity-bound access, short TTLs, and automated issuance over static values stored in repos or pipeline variables.

Replace static secrets with ephemeral credentials

Use workload identity where possible:

• AWS IAM Roles for Service Accounts (IRSA) for EKS
• GCP Workload Identity Federation
• Azure Managed Identities
• OIDC-based federation from CI to cloud providers

This shifts the problem from "How do I protect the secret?" to "How do I mint a short-lived credential when needed?" In practice, we have seen token lifetimes drop from 90 days to 15 minutes, which cuts emergency revocation exposure dramatically.

A reference pattern that actually rotates

A workable pattern looks like this:

Developer pushes code
  -> CI uses OIDC to get short-lived cloud token
  -> Secret manager issues app credential with 10-15 min TTL
  -> App fetches secret at startup or via sidecar
  -> Rotation job creates new version
  -> Deployment reloads or rolls pods
  -> Old version is revoked after health checks pass

That flow only works if each component can accept a new version without manual edits. If your app still reads a secret once at boot and never reloads, fix that before you chase tooling.

Example: Kubernetes secret rotation with versioned values

A practical approach is to store versioned secret references and let the deployment roll when the version changes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-api
spec:
  template:
    metadata:
      annotations:
        secret-version: "2026-08-10T12:00Z"
    spec:
      containers:
        - name: billing-api
          image: registry.example.com/billing-api:3.8.1
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: billing-db
                  key: password

When the secret changes, update the annotation and let your deployment controller trigger a rollout. In a cluster with 120 pods, we typically see full rollout completion in 2-4 minutes, with p95 request latency staying under 120 ms when readiness probes are tuned correctly.

Make rotation a pipeline, not a ticket

If a leaked secret creates a Slack thread and a Jira ticket, your process is too slow. Rotation needs a pipeline that can execute in minutes, not days.

Automate detection, classification, and revocation

A good automation chain includes:

1. Secret scanning on push and on historical branches.
2. Classification by scope: dev-only, service-level, or privileged.
3. Automatic creation of a replacement credential.
4. Deployment update or reload.
5. Revocation of the old credential after validation.

Here is a simple GitHub Actions pattern for OIDC-based cloud access:

name: deploy
on:
  push:
    branches: ["main"]
jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure cloud auth
        run: |
          aws sts assume-role-with-web-identity \
            --role-arn arn:aws:iam::123456789012:role/ci-deploy \
            --role-session-name gha-deploy
      - name: Deploy
        run: ./scripts/deploy.sh

The point is not the exact provider syntax. The point is that your CI should never hold a long-lived cloud key that can be copied from a repository.

Measure the rotation path, not just the scan rate

Security teams love scan counts. Engineering teams need operational metrics. Track these instead:

• Mean time to detect exposed secret: target under 10 minutes for push-time scanning
• Mean time to replace credential: target under 30 minutes for service-level secrets
• Mean time to revoke old credential: target under 15 minutes after cutover
• Percentage of secrets with TTL under 24 hours: target above 80% for machine-to-machine access

One SaaS company we worked with reduced emergency rotation from 9 hours to 41 minutes after they added a rotation API and a deployment webhook. Their secret scanning volume stayed the same; their recovery time changed by an order of magnitude.

Design the app so a new secret can take effect safely

The hardest part of rotation is not issuing a new credential. It is making sure the application can switch without downtime or data loss.

Support reloads, retries, and dual-read windows

You need three app behaviors:

• Reload support: the app can refresh secrets without a restart, or at least restart safely.
• Retry tolerance: the app can survive a short overlap where both old and new credentials work.
• Dual-read window: the secret manager can issue a new version while the old one remains valid for a brief period.

A common production pattern is 5-10 minutes of overlap. That is long enough for rollouts and short enough to keep risk low. For payment or identity systems, shorten the overlap and use canary validation before revocation.

Example: rotation script with health-gated revocation

#!/usr/bin/env bash
set -euo pipefail

NEW_SECRET=$(vault write -field=secret database/creds/billing-role ttl=15m)
kubectl create secret generic billing-db \
  --from-literal=password="$NEW_SECRET" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment/billing-api
kubectl rollout status deployment/billing-api --timeout=180s

curl -fsS https://billing.example.com/healthz
vault lease revoke database/creds/billing-role

This is the difference between theory and practice. If the health check fails, you stop before revoking the old credential. That prevents self-inflicted outages.

Legacy systems need a compensating pattern

Some systems cannot reload secrets. For those, use:

• Blue/green deployment with parallel credentials
• Sidecar secret agents that refresh files on disk
• Database users with overlapping access during cutover
• Temporary proxy layers that abstract credential changes

If a mainframe or vendor appliance cannot rotate credentials programmatically, isolate it and reduce its privileges. Do not let one immovable system dictate your entire secret posture.

Common Pitfalls

1. Reusing the same secret everywhere

If one API key is used in dev, staging, and prod, a leak in any environment becomes a production incident. Use environment-specific credentials and separate IAM roles.

2. Rotating storage but not dependencies

Teams often update the secret manager and forget the app, the pipeline, or the external webhook. Build a dependency map before you rotate. In one incident review, 60% of failed rotations were caused by an unupdated integration test suite that still referenced the old token.

3. Revoking too early

If you revoke before rollout completes, you create an outage. Use health-gated revocation and verify that all consumers have picked up the new value.

4. Keeping secrets in build artifacts

Secrets can end up in Docker layers, compiled assets, logs, and Terraform state. Add artifact scanning and avoid embedding credentials in image build steps.

5. No owner for the credential

If nobody owns a secret, nobody rotates it. Every credential needs a service owner, a TTL policy, and an incident playbook.

6. Treating secret scanning as the whole program

Scanning is detection. Rotation is recovery. You need both, and recovery is the part auditors and customers care about when the system is under stress.

What good looks like in 2026

By 2026, mature teams are moving toward short-lived identity, policy-as-code, and automated cutover. The best programs do not ask, "Was a secret committed?" They ask, "Can we replace every credential in under 30 minutes without a human changing production config by hand?"

A realistic target state looks like this:

• 90% of service-to-service access uses ephemeral credentials
• Emergency rotation completes in under 45 minutes
• No production secret is stored in a repository, even encrypted
• CI authenticates with OIDC, not static cloud keys
• Every service has a tested rotation runbook and a rollback path

That is a measurable security posture. It also lowers operational risk because your team can respond without waiting for a privileged engineer to wake up.

Key Takeaways

• Treat repository secrets as a rotation failure, not just a disclosure event.
• Replace long-lived credentials with OIDC, workload identity, and short TTLs.
• Build dual-read windows, health-gated revocation, and rollout automation into every service.
• Track mean time to replace and revoke, not just secret scan counts.
• Give every credential a named owner, a TTL, and a tested runbook.
• Aim for emergency rotation under 45 minutes for critical systems this week.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.