Dynamic DB credentials: lease TTLs vs connection pool failures

TL;DR — Dynamic database credentials solve static-password sprawl, but the failure mode is predictable: your app’s connection pool keeps using sessions authenticated with a user whose lease has expired or been revoked. The single most likely fix is to align broker lease TTL, pool max connection lifetime, and credential refresh timing so pooled connections are retired before the database user disappears. Reading time: ~7 min

What it is and where it sits

A secrets broker can create database credentials on demand instead of handing your app a long-lived shared password. The broker talks to the database using an admin-capable account, creates a DB user/role with a TTL, returns the username/password plus lease metadata to the caller, and later renews or revokes that lease.

In a typical stack, this replaces one of these patterns:

• a static password in environment variables
• a shared application DB user stored in CI/CD secrets
• manually rotated credentials copied into config files

Architecturally, it sits between your workload and the database, but not on the query path. It is a control-plane dependency, not usually a data-plane proxy. Your app still connects directly to Postgres/MySQL using ordinary drivers. The broker issues the credential; your pool uses it until you rotate.

[App Process] ---- 1. request dynamic DB creds ----> [Secrets Broker]
[App Process] <--- 2. username,password,lease TTL --- [Secrets Broker]
[App Process] ---- 3. open pooled DB conns -------> [Postgres]
[Secrets Broker] ---- 4. create/revoke DB user ----> [Postgres]
[HTTP Request]
|
v
[App Process] ---- 5. query using pooled conn ----> [Postgres]

From a Privileged Access Management angle, the point is not just convenience. It narrows blast radius:

• each workload can get a distinct DB principal
• leases expire automatically
• revocation is central and auditable
• the app no longer needs standing privileged knowledge of a permanent password

The catch: database sessions and application pools have their own lifecycles. A lease expiring in the broker does not magically reconfigure your pool.

How it actually works

Walk one realistic example: a Go API uses a broker to fetch Postgres credentials with a 1 hour lease. The app uses pgxpool. The broker creates a role like v-app-readwrite-7f3c2 with VALID UNTIL '2026-10-30 13:00:00+00' or equivalent expiry semantics, and returns JSON including lease duration.

Step 1: App fetches credentials

Typical response shape from a broker API looks like this:

{
  "username": "v-app-readwrite-7f3c2",
  "password": "s3cr3t-temporary",
  "lease_id": "database/creds/app/9xK...",
  "lease_duration": 3600,
  "renewable": true
}

Your app builds a DSN from that and starts its pool.

Step 2: Pool opens connections

Suppose the pool is configured like this:

• max pool size: 20
• idle timeout: 30m
• max connection lifetime: unlimited
• health checks: default

At startup, the app opens a few connections. More are added under load. Every connection authenticates successfully because the DB user exists and the password is valid.

Step 3: Traffic is fine until the lease boundary

For the next hour, requests succeed. Existing sessions keep working depending on database behavior and revocation method:

• If the broker only prevents new logins by setting an expiry on the DB user, existing sessions may continue until disconnected.
• If the broker also revokes privileges or terminates sessions, in-flight or next-use failures happen immediately.

This distinction matters. Experienced teams test both cases because “credential expired” does not always mean “all current queries fail now.”

Step 4: Lease expires or is revoked

At T+3600s, one of several things happens:

1. New connections fail authentication.
2. Existing pooled connections continue for a while, masking the problem.
3. Existing pooled connections are killed, causing sudden query errors.

What your app sees depends on timing. Common symptoms:

FATAL: password authentication failed for user "v-app-readwrite-7f3c2"
SQLSTATE 28P01

Or, if sessions are terminated:

FATAL: terminating connection due to administrator command
SQLSTATE 57P01

Or at the pool layer:

acquire connection: context deadline exceeded

The nasty version is partial failure: half the requests succeed because they hit old live sessions; the rest fail because the pool tries to open replacement connections with expired credentials.

Step 5: Why renewal alone often doesn’t fix it

If your app renews the lease but does not update the pool’s DSN or recycle existing connections, nothing changes. The pool still holds connections authenticated with the old username/password. If the broker rotates by issuing an entirely new username, renewal of the old lease is not even the right model; you need re-fetch + pool swap.

Concrete rule: if credentials can change, your app needs a credential refresh path and a connection retirement path.

A safe timing model is:

• lease TTL: 60m
• app refreshes at: 45m
• pool max connection lifetime: 40m
• overlap/grace period: 10-15m where old and new creds both work, if your broker/database pattern allows it

That way, by the time the old lease expires, most or all pooled connections created with old creds have already been drained.

When to use it (and when not to)

You probably don’t need this if all of the following are true:

• one or two services only
• no per-service audit requirement
• no practical revocation requirement faster than your normal deploy cycle
• your app cannot reload DB credentials without a restart

Trade-offs

Benefit and cost come together here.

• Benefit: no long-lived shared DB password in app config
  Cost: you now operate a broker integration, lease renewal logic, and failure handling.

• Benefit: revocation is fast and centralized
  Cost: revocation can cause production incidents if pool lifetime exceeds lease lifetime or if sessions are terminated aggressively.

• Benefit: per-workload identities improve audit trails
  Cost: databases accumulate lots of ephemeral users/roles unless cleanup is reliable. Catalog bloat and admin clutter are real concerns.

• Benefit: reduced blast radius after secret exfiltration
  Cost: shorter TTLs increase login churn, broker traffic, and DB role creation/deletion overhead.

• Benefit: better PAM posture
  Cost: tighter coupling to broker semantics. “Renew” vs “re-issue new creds” is not interchangeable, and implementations vary.

• Benefit: automatic expiry for ephemeral workloads
  Cost: bad fit for apps with sticky, long-lived pools unless you explicitly tune connection lifetime.

Latency usually is not the main cost on the query path because the broker is not in-band for every query. The real latency cost shows up at startup, refresh, and reconnect storms.

In practice

Example 1: Postgres role TTL and what to inspect

If you want to understand the database side, inspect role expiry and active sessions directly.

⚠️ Terminating sessions will drop in-flight transactions and can cause user-visible errors. Run the inspection query first; run the termination query only during a planned test or incident response.

-- Check when a dynamic role expires
SELECT rolname, rolvaliduntil
FROM pg_authid
WHERE rolname LIKE 'v-app-%';

-- See active sessions for one dynamic user
SELECT pid, usename, application_name, client_addr, state, backend_start
FROM pg_stat_activity
WHERE usename = 'v-app-readwrite-7f3c2';

-- Force disconnect sessions for that user
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE usename = 'v-app-readwrite-7f3c2';

This shows whether your broker is only expiring future logins or whether active sessions are still carrying traffic. The gotcha: pg_authid is privileged; use an admin session, not the application user.

Example 2: Go pgxpool with credential refresh and bounded connection lifetime

package main

import (
    "context"
    "fmt"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"
)

type DBCreds struct {
    Username      string
    Password      string
    Host          string
    Port          int
    Database      string
    LeaseDuration time.Duration
}

func fetchCreds(ctx context.Context) (DBCreds, error) {
    // Replace with your broker API call.
    return DBCreds{
        Username:      "v-app-readwrite-7f3c2",
        Password:      "temporary-password",
        Host:          "db.internal",
        Port:          5432,
        Database:      "appdb",
        LeaseDuration: 60 * time.Minute,
    }, nil
}

func newPool(ctx context.Context, c DBCreds) (*pgxpool.Pool, error) {
    dsn := fmt.Sprintf("postgres://%s:%s@%s:%d/%s", c.Username, c.Password, c.Host, c.Port, c.Database)
    cfg, err := pgxpool.ParseConfig(dsn)
    if err != nil {
        return nil, err
    }
    cfg.MaxConns = 20
    cfg.MaxConnLifetime = 40 * time.Minute
    cfg.MaxConnIdleTime = 10 * time.Minute
    return pgxpool.NewWithConfig(ctx, cfg)
}

func main() {
    ctx := context.Background()
    creds, _ := fetchCreds(ctx)
    pool, _ := newPool(ctx, creds)
    defer pool.Close()

    refreshAfter := 45 * time.Minute
    ticker := time.NewTicker(refreshAfter)
    defer ticker.Stop()

    for range ticker.C {
        nextCreds, err := fetchCreds(ctx)
        if err != nil {
            continue
        }
        nextPool, err := newPool(ctx, nextCreds)
        if err != nil {
            continue
        }

        oldPool := pool
        pool = nextPool

        go func() {
            time.Sleep(2 * time.Minute)
            oldPool.Close()
        }()
    }
}

This pattern swaps the whole pool after fetching fresh credentials and gives old requests a short drain period. The gotcha: if your app stores the pool in multiple places, a partial swap creates heisenbugs; keep one shared pool reference behind an accessor.

Example 3: HikariCP settings to stop stale DB sessions outliving leases

# application.properties / hikari config
spring.datasource.hikari.maximumPoolSize=20
spring.datasource.hikari.minimumIdle=2
spring.datasource.hikari.maxLifetime=2400000
spring.datasource.hikari.idleTimeout=600000
spring.datasource.hikari.keepaliveTime=300000
spring.datasource.hikari.connectionTimeout=30000

Here maxLifetime=2400000 sets 40 minutes, which is intentionally lower than a 60 minute credential lease. The gotcha: if you set maxLifetime longer than the broker TTL, the pool will happily try to reuse connections tied to dead credentials and failures will cluster at reconnect time.

Example 4: Quick diagnosis from the app host

export PGPASSWORD='temporary-password'
psql "host=db.internal port=5432 dbname=appdb user=v-app-readwrite-7f3c2 sslmode=require" -c 'select now();'
echo $?

Expected success:

              now
-------------------------------
 2026-10-30 12:14:03.221145+00
(1 row)

0

After expiry or revocation:

psql: error: connection to server at "db.internal" (10.0.4.12), port 5432 failed: FATAL:  password authentication failed for user "v-app-readwrite-7f3c2"
2

This isolates “bad credentials” from “pool bug” quickly. The gotcha: psql exit code 2 is generic connection failure; read stderr for the actual auth or network cause.

Further reading

• PostgreSQL Documentation: Role Attributes
• PostgreSQL Documentation: pg_stat_activity
• HikariCP Configuration Knobs, Baby
• pgx package documentation for pgxpool
• The OWASP Secrets Management Cheat Sheet

This article was written by an AI system and published pending human review. Verify anything you intend to act on.