Approval-based sudo via ChatOps with durable audit logging
For developers wiring privileged access into a ChatOps workflow without losing the audit trail when chat messages expire. You will set up approval-gated elevation that writes immutable request, approval, execution, and session records to a database and local system logs, then verify the full chain end to end.
TL;DR — Build the approval gate outside the chat platform: chat triggers a webhook, the webhook creates a request record in Postgres, a second approver signs off, and only then does a short-lived sudo rule get activated for the target user. The key fix for "we approved it in chat but have no audit trail now" is to persist every state change and command/session record to durable storage at request time, not by scraping chat history later. Reading time: ~5 min
Goal
When you finish, a developer can request temporary privileged access from chat, a separate approver can approve it, the target host grants sudo only for the approved window, and the full audit trail exists in Postgres and system logs even if the chat thread is deleted or expires.
Prerequisites
- A Linux target host with
sudo >= 1.9andsystemd— check withsudo -V | head -1andsystemctl --version | head -1 - Root or equivalent admin access on the target host
- PostgreSQL 15 or newer reachable from the webhook service — check with
psql --version - A webhook-capable chat platform or bot framework that can POST JSON to your service
- A small service host with
python >= 3.11— check withpython3 --version curl,jq, andopensslinstalled on the service host- TLS certificate for the webhook endpoint, or a reverse proxy that already terminates TLS
- The Unix username to elevate, the approver identity source, and the exact command scope you want to allow, for example
/usr/bin/systemctl restart myapp
Steps
Step 1: Create the durable audit schema
Run these commands on the PostgreSQL host or anywhere with psql access:
psql "$DATABASE_URL" <<'SQL'
create table if not exists pam_requests (
id uuid primary key,
requested_at timestamptz not null default now(),
requester_chat_id text not null,
requester_name text not null,
target_unix_user text not null,
host text not null,
command_pattern text not null,
justification text not null,
status text not null check (status in ('pending','approved','denied','expired','executed')),
expires_at timestamptz,
chat_thread_ref text,
metadata jsonb not null default '{}'::jsonb
);
create table if not exists pam_approvals (
id bigserial primary key,
request_id uuid not null references pam_requests(id) on delete cascade,
decided_at timestamptz not null default now(),
approver_chat_id text not null,
approver_name text not null,
decision text not null check (decision in ('approved','denied')),
reason text,
unique (request_id, approver_chat_id)
);
create table if not exists pam_executions (
id bigserial primary key,
request_id uuid not null references pam_requests(id) on delete cascade,
executed_at timestamptz not null default now(),
host text not null,
unix_user text not null,
command text not null,
exit_code integer,
session_id text,
log_ref text
);
SQL
You should see CREATE TABLE or NOTICE: relation ... already exists, skipping.
Step 2: Install a restricted sudoers rule and I/O logging
⚠️ A bad sudoers file can lock out admin access. Use
visudo -cfbefore enabling any file under/etc/sudoers.d.
On the target host, create a command alias and enable I/O logging to syslog and disk:
sudo install -d -m 0750 /var/log/sudo-io
sudo tee /etc/sudoers.d/chatops-pam >/dev/null <<'EOF'
Defaults logfile="/var/log/sudo.log"
Defaults log_output,log_input,iolog_dir="/var/log/sudo-io"
Defaults! /usr/bin/systemctl syslog=authpriv
Cmnd_Alias CHATOPS_MYAPP = /usr/bin/systemctl restart myapp, /usr/bin/systemctl status myapp
%chatops_elevated ALL=(root) NOPASSWD: CHATOPS_MYAPP
EOF
sudo visudo -cf /etc/sudoers.d/chatops-pam
sudo groupadd -f chatops_elevated
Success is a single line like:
/etc/sudoers.d/chatops-pam: parsed OK
Step 3: Add a TTL-based elevation script on the target host
Create a script that adds the user to the sudo-enabled group, then removes them after the approved duration:
sudo tee /usr/local/sbin/chatops-elevate >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
USER_NAME="$1"
TTL_MINUTES="$2"
id "$USER_NAME" >/dev/null 2>&1
usermod -aG chatops_elevated "$USER_NAME"
systemd-run --unit="chatops-deelevate-${USER_NAME}" --on-active="${TTL_MINUTES}m" /usr/sbin/gpasswd -d "$USER_NAME" chatops_elevated
logger -t chatops-pam "elevated user=${USER_NAME} ttl=${TTL_MINUTES}m"
EOF
sudo chmod 0750 /usr/local/sbin/chatops-elevate
You should see no output and exit code 0 from echo $?.
Step 4: Start the approval webhook service
On the service host, create the app and install dependencies:
python3 -m venv /opt/chatops-pam-venv
/opt/chatops-pam-venv/bin/pip install flask psycopg[binary] gunicorn
install -d /opt/chatops-pam
cat >/opt/chatops-pam/app.py <<'PY'
import os, uuid, json, subprocess
from datetime import datetime, timedelta, timezone
from flask import Flask, request, jsonify
import psycopg
app = Flask(__name__)
DB = os.environ["DATABASE_URL"]
SSH_HOST = os.environ["TARGET_HOST"]
DEFAULT_TTL = int(os.environ.get("DEFAULT_TTL_MIN", "15"))
def q(sql, args=(), fetch=False):
with psycopg.connect(DB) as conn:
with conn.cursor() as cur:
cur.execute(sql, args)
if fetch:
return cur.fetchall()
conn.commit()
@app.post('/request')
def create_request():
d = request.get_json(force=True)
rid = str(uuid.uuid4())
q("insert into pam_requests (id, requester_chat_id, requester_name, target_unix_user, host, command_pattern, justification, status, expires_at, chat_thread_ref, metadata) values (%s,%s,%s,%s,%s,%s,%s,'pending',%s,%s,%s)", (
rid, d['requester_chat_id'], d['requester_name'], d['target_unix_user'], SSH_HOST, d['command_pattern'], d['justification'], datetime.now(timezone.utc)+timedelta(minutes=DEFAULT_TTL), d.get('chat_thread_ref'), json.dumps(d)))
return jsonify({"request_id": rid, "status": "pending"}), 201
@app.post('/approve')
def approve():
d = request.get_json(force=True)
req = q("select requester_chat_id,target_unix_user,status from pam_requests where id=%s", (d['request_id'],), True)
if not req: return jsonify({"error":"request_not_found"}), 404
requester_chat_id, target_unix_user, status = req[0]
if requester_chat_id == d['approver_chat_id']:
return jsonify({"error":"self_approval_forbidden"}), 403
if status != 'pending':
return jsonify({"error":"request_not_pending","status":status}), 409
q("insert into pam_approvals (request_id, approver_chat_id, approver_name, decision, reason) values (%s,%s,%s,'approved',%s)", (d['request_id'], d['approver_chat_id'], d['approver_name'], d.get('reason')))
subprocess.run(["ssh", "root@" + SSH_HOST, "/usr/local/sbin/chatops-elevate", target_unix_user, str(DEFAULT_TTL)], check=True)
q("update pam_requests set status='approved' where id=%s", (d['request_id'],))
return jsonify({"request_id": d['request_id'], "status": "approved"})
PY
cat >/etc/systemd/system/chatops-pam.service <<'EOF'
[Unit]
Description=ChatOps PAM webhook
After=network.target
[Service]
Environment=DATABASE_URL=postgresql://pam_user:REPLACE_ME@db.example.internal/pam
Environment=TARGET_HOST=app01.example.internal
Environment=DEFAULT_TTL_MIN=15
WorkingDirectory=/opt/chatops-pam
ExecStart=/opt/chatops-pam-venv/bin/gunicorn -b 127.0.0.1:8000 app:app
Restart=always
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now chatops-pam.service
sudo systemctl status --no-pager chatops-pam.service
You should see Active: active (running).
Step 5: Put TLS in front of the webhook
If you already have nginx, add this server block and reload:
server {
listen 443 ssl http2;
server_name pam-webhook.example.com;
ssl_certificate /etc/letsencrypt/live/pam-webhook.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/pam-webhook.example.com/privkey.pem;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
sudo nginx -t && sudo systemctl reload nginx
curl -I https://pam-webhook.example.com/request
You should see a status line like HTTP/2 405 because /request only accepts POST.
Step 6: Wire the chat action to the webhook payloads
From your chat platform, send these exact JSON shapes to your webhook. First, create a request:
curl -sS -X POST https://pam-webhook.example.com/request \
-H 'content-type: application/json' \
-d '{
"requester_chat_id":"u12345",
"requester_name":"Alex Dev",
"target_unix_user":"alex",
"command_pattern":"/usr/bin/systemctl restart myapp",
"justification":"rollback failed; restart required",
"chat_thread_ref":"room42/thread99"
}' | jq
Then approve it as a different user:
curl -sS -X POST https://pam-webhook.example.com/approve \
-H 'content-type: application/json' \
-d '{
"request_id":"REPLACE_WITH_REQUEST_ID",
"approver_chat_id":"u67890",
"approver_name":"Sam Lead",
"reason":"incident INC-1042"
}' | jq
You should see JSON with "status": "pending" from the first call and "status": "approved" from the second.
Verify it works
Run the full chain checks.
View as text
[Requester Chat] ---- 1. POST /request ----> [Webhook] [Approver Chat] ---- 2. POST /approve ---> [Webhook] [Webhook] ---- 3. insert rows -----------> [Postgres] [Webhook] ---- 4. ssh elevate -----------> [Target Host] [Target Host] ---- 5. sudo + I/O logs ---> [Local Logs]
psql "$DATABASE_URL" -x -c "select id,status,requester_name,target_unix_user,host from pam_requests order by requested_at desc limit 1"
psql "$DATABASE_URL" -x -c "select request_id,approver_name,decision from pam_approvals order by decided_at desc limit 1"
ssh alex@app01.example.internal 'id && sudo /usr/bin/systemctl status myapp; echo exit_code=$?'
sudo journalctl -t chatops-pam -n 5 --no-pager
sudo tail -n 20 /var/log/sudo.log
Expected results:
pam_requests.statusisapprovedpam_approvals.decisionisapprovedidoutput on the target host includeschatops_elevatedsudo /usr/bin/systemctl status myappreturnsexit_code=0/var/log/sudo.logcontains the invoking user and command
A healthy curl -I to the webhook looks like:
HTTP/2 405
server: nginx/1.24.0
content-type: text/html; charset=utf-8
allow: OPTIONS, POST
Common pitfalls
Self-approval is not blocked
Mistake: you approve with the same chat identity that created the request.
Symptom:
{"error":"self_approval_forbidden"}
Fix: send /approve with a different approver_chat_id than requester_chat_id.
The user was approved but still cannot sudo
Mistake: the user already had a long-lived shell and group membership was not refreshed in that session.
Symptom:
alex is not in the sudoers file.
Fix: open a new SSH session after approval, or run newgrp chatops_elevated in the existing shell.
SSH from webhook to target host fails
Mistake: the service host has no root SSH key or host key trust for the target host.
Symptom:
subprocess.CalledProcessError: Command '['ssh', 'root@app01.example.internal', ...]' returned non-zero exit status 255.
Fix: install a dedicated SSH key for the webhook service and pre-seed known_hosts with ssh-keyscan app01.example.internal >> /root/.ssh/known_hosts.
Sudoers syntax is wrong
Mistake: editing /etc/sudoers.d/chatops-pam without validating it.
Symptom:
/etc/sudoers.d/chatops-pam: syntax error near line 3
Fix: run sudo visudo -cf /etc/sudoers.d/chatops-pam, correct the line, and test again before relying on it.
You stored only the chat message link
Mistake: recording chat_thread_ref but not the request body, approver identity, and execution record in Postgres.
Symptom: the chat thread expires and you cannot prove who approved what.
Fix: keep the metadata JSON, pam_approvals, and pam_executions rows as the source of truth; treat chat links as convenience only.
This article was written by an AI system and published pending human review. Verify anything you intend to act on.
Have a project in mind?
Get an instant AI price estimate for it, or talk directly to our team.
One email a month on what we learn building with AI