Version prompts and model config to trace AI quality regressions

TL;DR — Put prompts and model settings in versioned files, generate a deterministic config hash for every release, and log that hash with each model call. The single biggest fix is to stop storing prompt text and parameters inline in app code or dashboards; move them into files committed to Git and stamp every request with a version ID. Reading time: ~5 min

Goal

When you finish, every production model call will carry a traceable prompt version and model-config version, your repo will contain the exact prompt/config files used for each release, and you will be able to map a quality regression to a specific Git commit or config hash from logs alone.

Prerequisites

• Git access to the application repo and permission to merge to the deployment branch
• Node.js >= 20 if you want the example scripts; check with:

node --version

• jq >= 1.6 for JSON normalization; check with:

jq --version

• A place to store application logs or traces that include request metadata
• Your current model name(s), temperature/top_p/max_tokens, tool settings, system prompt, and any retrieval settings written down before you start
• CI access to add a validation step

Steps

Step 1: Create a versioned repo layout for prompts and model config

Create directories and starter files exactly like this:

mkdir -p ai/prompts ai/config scripts
cat > ai/prompts/support_reply.v1.md <<'EOF'
You are a support assistant for Acme.
Answer using the product docs only.
If the docs do not answer the question, say: "I don't know based on the docs provided."
EOF

cat > ai/config/support_reply.v1.json <<'EOF'
{
  "model": "gpt-4.1-mini",
  "temperature": 0.2,
  "top_p": 1,
  "max_output_tokens": 600,
  "reasoning_effort": "medium",
  "tools": [],
  "response_format": { "type": "text" }
}
EOF

Success looks like this:

$ find ai -maxdepth 2 -type f | sort
ai/config/support_reply.v1.json
ai/prompts/support_reply.v1.md

Step 2: Add a deterministic hash script for config + prompt content

Create a script that normalizes JSON and hashes the prompt plus config. This gives you an immutable version ID even if file ordering changes.

cat > scripts/hash-ai-config.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
PROMPT_FILE="$1"
CONFIG_FILE="$2"
TMP_JSON="$(mktemp)"
jq -S . "$CONFIG_FILE" > "$TMP_JSON"
{
  printf 'prompt_file=%s\n' "$PROMPT_FILE"
  cat "$PROMPT_FILE"
  printf '\nconfig_file=%s\n' "$CONFIG_FILE"
  cat "$TMP_JSON"
} | sha256sum | awk '{print $1}'
rm -f "$TMP_JSON"
EOF
chmod +x scripts/hash-ai-config.sh
./scripts/hash-ai-config.sh ai/prompts/support_reply.v1.md ai/config/support_reply.v1.json

Success is a 64-character SHA-256 value, for example:

$ ./scripts/hash-ai-config.sh ai/prompts/support_reply.v1.md ai/config/support_reply.v1.json
7d7a1d4d5f7f4f6c2d8b5d1d8b0f0a4b9d5a8e7c1f3e2d4c6b7a8e9f0a1b2c3d

Step 3: Add a manifest that maps a logical task name to exact files

Create a manifest so app code refers to one logical task and resolves to concrete versioned files.

cat > ai/manifest.json <<'EOF'
{
  "support_reply": {
    "prompt_file": "ai/prompts/support_reply.v1.md",
    "config_file": "ai/config/support_reply.v1.json"
  }
}
EOF

jq . ai/manifest.json

Success is valid JSON output with the exact file paths.

Step 4: Load the manifest in app code and attach version metadata to every request

Use code like this in your app. It reads the prompt/config from disk, computes the hash, and logs metadata before calling the model.

cat > scripts/example-loader.mjs <<'EOF'
import fs from 'node:fs';
import crypto from 'node:crypto';

const manifest = JSON.parse(fs.readFileSync('ai/manifest.json', 'utf8'));
const spec = manifest['support_reply'];
const prompt = fs.readFileSync(spec.prompt_file, 'utf8');
const config = JSON.parse(fs.readFileSync(spec.config_file, 'utf8'));
const normalizedConfig = JSON.stringify(Object.keys(config).sort().reduce((o, k) => (o[k] = config[k], o), {}));
const versionId = crypto.createHash('sha256').update(`prompt_file=${spec.prompt_file}\n${prompt}\nconfig_file=${spec.config_file}\n${normalizedConfig}`).digest('hex');

const requestMeta = {
  ai_feature: 'support_reply',
  ai_prompt_file: spec.prompt_file,
  ai_config_file: spec.config_file,
  ai_version_id: versionId,
  ai_model: config.model
};

console.log(JSON.stringify(requestMeta, null, 2));
EOF
node scripts/example-loader.mjs

Success looks like:

{
  "ai_feature": "support_reply",
  "ai_prompt_file": "ai/prompts/support_reply.v1.md",
  "ai_config_file": "ai/config/support_reply.v1.json",
  "ai_version_id": "7d7a1d4d5f7f4f6c2d8b5d1d8b0f0a4b9d5a8e7c1f3e2d4c6b7a8e9f0a1b2c3d",
  "ai_model": "gpt-4.1-mini"
}

Step 5: Log the version ID with the model response and user-visible output

Add these exact fields to your structured logs for every inference:

{
  "event": "ai_inference",
  "ai_feature": "support_reply",
  "ai_prompt_file": "ai/prompts/support_reply.v1.md",
  "ai_config_file": "ai/config/support_reply.v1.json",
  "ai_version_id": "7d7a1d4d5f7f4f6c2d8b5d1d8b0f0a4b9d5a8e7c1f3e2d4c6b7a8e9f0a1b2c3d",
  "model": "gpt-4.1-mini",
  "request_id": "req_01K2...",
  "user_id": "12345",
  "latency_ms": 842,
  "output_rating": null
}

Success means a single log line lets you identify the exact prompt/config used for that output.

Step 6: Add CI validation so unversioned changes fail the build

Create a validation script that rejects non-versioned filenames and invalid JSON.

cat > scripts/validate-ai-assets.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
find ai/prompts -type f | grep -Ev '\.v[0-9]+\.(md|txt)$' && { echo 'Unversioned prompt filename found'; exit 1; } || true
find ai/config -type f | grep -Ev '\.v[0-9]+\.json$' && { echo 'Unversioned config filename found'; exit 1; } || true
find ai/config -name '*.json' -print0 | xargs -0 -n1 jq empty
jq empty ai/manifest.json
EOF
chmod +x scripts/validate-ai-assets.sh
./scripts/validate-ai-assets.sh

Success is no output and exit code 0:

$ ./scripts/validate-ai-assets.sh
$ echo $?
0

If it fails, you will see output shaped like:

$ ./scripts/validate-ai-assets.sh
ai/prompts/support_reply.md
Unversioned prompt filename found
$ echo $?
1

Step 7: Release a change by creating new files, not editing old ones

When you change behavior, create v2 files and update only the manifest pointer.

cp ai/prompts/support_reply.v1.md ai/prompts/support_reply.v2.md
cp ai/config/support_reply.v1.json ai/config/support_reply.v2.json
perl -0pi -e 's/support_reply\.v1\.md/support_reply.v2.md/' ai/manifest.json
perl -0pi -e 's/support_reply\.v1\.json/support_reply.v2.json/' ai/manifest.json
git add ai/ scripts/
git commit -m "ai: support_reply v2 prompt and config"

Success means git diff HEAD~1..HEAD -- ai/manifest.json ai/prompts ai/config shows new files plus a one-line manifest change.

⚠️ If you overwrite v1 files instead of creating v2, you destroy traceability for old outputs. Do not rewrite historical prompt/config files after they have been used in any environment you care about.

Verify it works

Run these checks end to end:

./scripts/hash-ai-config.sh ai/prompts/support_reply.v2.md ai/config/support_reply.v2.json
node scripts/example-loader.mjs
git log --oneline -- ai/manifest.json ai/prompts ai/config | head

Expected results:

• The hash script returns one stable SHA-256 value for the current prompt/config pair.
• The loader prints ai_prompt_file, ai_config_file, and ai_version_id.
• git log shows the commit that introduced the current versioned files.

If you have centralized logs, query for a bad request ID and confirm the metadata is present. A healthy log record shape is:

{
  "request_id": "req_01K2...",
  "ai_feature": "support_reply",
  "ai_version_id": "<64-char hash>",
  "ai_prompt_file": "ai/prompts/support_reply.v2.md",
  "ai_config_file": "ai/config/support_reply.v2.json"
}

Common pitfalls

Editing prompt text inline in application code

Mistake: the system prompt or parameters live in a source file next to business logic. Symptom: git blame shows code changes, but logs cannot tell which prompt text produced a bad answer. Fix: move prompt text and model settings into ai/prompts/*.vN.md and ai/config/*.vN.json, then log ai_version_id.

Reusing the same filename for changed behavior

Mistake: support_reply.v1.md is edited in place after release. Symptom: old logs point to v1, but the file contents in the repo no longer match historical behavior. Fix: create support_reply.v2.md and support_reply.v2.json; never mutate previously used versioned files.

Hashing raw JSON without normalization

Mistake: the version ID is computed from pretty-printed JSON as-is. Symptom: the hash changes when someone reorders keys or changes indentation, even though runtime behavior is identical. Fix: normalize with jq -S . before hashing.

Logging only the model name

Mistake: logs contain model=gpt-4.1-mini and nothing else. Symptom: regressions cannot be tied to prompt edits, token limits, tool changes, or reasoning settings. Fix: log ai_prompt_file, ai_config_file, ai_version_id, and the resolved model name on every inference.

Changing retrieval or tool settings outside the versioned config

Mistake: tool lists, document filters, or top-k retrieval values are configured elsewhere. Symptom: outputs change but the prompt/config hash stays the same, so your trace is incomplete. Fix: put retrieval and tool parameters in the same versioned JSON file or generate a second hash and log both.

Depending on a provider dashboard as the source of truth

Mistake: prompt templates or parameters are edited directly in a vendor UI. Symptom: production behavior changes without a Git commit, code review, or reproducible artifact. Fix: keep the canonical prompt/config in Git and have deployment sync from repo to runtime, not the other way around.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.