Build a GDPR DSAR export pipeline across multiple services

TL;DR — Build one repeatable export job that accepts a stable subject identifier, pulls data from each service into JSON files, writes a manifest with hashes, packages everything into a timestamped archive, and stores it in a short-lived private location. The most common failure is mismatched identity keys across services; fix that first by defining one canonical lookup input and recording every identifier mapping in the manifest. Reading time: ~5 min

Goal

When you finish, you will have a command-line DSAR export job that gathers a single user’s data from multiple services, writes a structured export bundle to disk, hashes and archives it, and uploads it to private object storage with a time-limited download URL.

Prerequisites

• Access to each service that stores personal data for the subject: database read access, API credentials, or both
• A Linux/macOS shell with these tools installed:
  • bash >= 5 — check with bash --version
  • jq >= 1.6 — check with jq --version
  • curl >= 8 — check with curl --version
  • psql >= 15 if you query PostgreSQL — check with psql --version
  • zip >= 3.0 and sha256sum — check with zip -v | head -1 and sha256sum --version | head -1
  • AWS CLI v2 if you upload to S3-compatible storage — check with aws --version
• Read-only credentials for each source system, for example:
  • APP_DB_URL for the main PostgreSQL database
  • BILLING_API_BASE and BILLING_API_TOKEN
  • SUPPORT_API_BASE and SUPPORT_API_TOKEN
  • S3_BUCKET and AWS credentials with s3:PutObject and s3:PutObjectTagging
• One canonical subject input you can reliably start from, usually email or internal user ID
• A private working directory on an encrypted host with enough free space for the export

Steps

Step 1: Create the export workspace and environment file

mkdir -p ~/dsar-export/bin ~/dsar-export/sql ~/dsar-export/out
cd ~/dsar-export
cat > .env <<'EOF'
APP_DB_URL='postgresql://readonly:REDACTED@db.example.internal:5432/app'
BILLING_API_BASE='https://billing.example.internal/api/v1'
BILLING_API_TOKEN='REDACTED'
SUPPORT_API_BASE='https://support.example.internal/api/v1'
SUPPORT_API_TOKEN='REDACTED'
S3_BUCKET='private-dsar-exports'
AWS_REGION='eu-west-1'
EOF
chmod 600 .env

You should see no output, and ls -l .env should show permissions like -rw-------.

Step 2: Define the canonical identity lookup query

Create one SQL query that resolves the subject from your canonical input and returns all known identifiers used by downstream systems.

cat > sql/identity.sql <<'SQL'
WITH subject AS (
  SELECT id, email, created_at
  FROM users
  WHERE email = :'email'
)
SELECT json_build_object(
  'user_id', s.id,
  'email', s.email,
  'created_at', s.created_at,
  'billing_customer_id', bc.customer_id,
  'support_contact_id', sc.contact_id
)
FROM subject s
LEFT JOIN billing_customers bc ON bc.user_id = s.id
LEFT JOIN support_contacts sc ON sc.user_id = s.id;
SQL

You should have a file at sql/identity.sql with one JSON-producing row.

Step 3: Create the export script

cat > bin/export-dsar.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail

if [ $# -ne 1 ]; then
  echo "usage: $0 <email>" >&2
  exit 64
fi

EMAIL="$1"
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
source "$ROOT/.env"

TS="$(date -u +%Y%m%dT%H%M%SZ)"
SAFE_EMAIL="$(printf '%s' "$EMAIL" | tr '@' '_' | tr -cd '[:alnum:]_.-')"
WORKDIR="$ROOT/out/${TS}_${SAFE_EMAIL}"
mkdir -p "$WORKDIR/raw"

IDENTITY_JSON="$WORKDIR/raw/identity.json"
psql "$APP_DB_URL" -X -v ON_ERROR_STOP=1 -v email="$EMAIL" -tA -f "$ROOT/sql/identity.sql" > "$IDENTITY_JSON"

if [ ! -s "$IDENTITY_JSON" ] || [ "$(cat "$IDENTITY_JSON")" = "" ]; then
  echo "identity lookup returned no data for $EMAIL" >&2
  exit 66
fi

USER_ID="$(jq -r '.user_id // empty' "$IDENTITY_JSON")"
BILLING_CUSTOMER_ID="$(jq -r '.billing_customer_id // empty' "$IDENTITY_JSON")"
SUPPORT_CONTACT_ID="$(jq -r '.support_contact_id // empty' "$IDENTITY_JSON")"

psql "$APP_DB_URL" -X -v ON_ERROR_STOP=1 -tA -c "SELECT coalesce(json_agg(t), '[]'::json) FROM (SELECT id, email, name, created_at, updated_at FROM users WHERE id = '$USER_ID') t" > "$WORKDIR/raw/app_user.json"
psql "$APP_DB_URL" -X -v ON_ERROR_STOP=1 -tA -c "SELECT coalesce(json_agg(t), '[]'::json) FROM (SELECT id, user_id, created_at, ip_address, user_agent FROM login_events WHERE user_id = '$USER_ID' ORDER BY created_at DESC) t" > "$WORKDIR/raw/login_events.json"
psql "$APP_DB_URL" -X -v ON_ERROR_STOP=1 -tA -c "SELECT coalesce(json_agg(t), '[]'::json) FROM (SELECT id, user_id, status, total_cents, currency, created_at FROM orders WHERE user_id = '$USER_ID' ORDER BY created_at DESC) t" > "$WORKDIR/raw/orders.json"

if [ -n "$BILLING_CUSTOMER_ID" ]; then
  curl -fsS -H "Authorization: Bearer $BILLING_API_TOKEN" "$BILLING_API_BASE/customers/$BILLING_CUSTOMER_ID" | jq . > "$WORKDIR/raw/billing_customer.json"
  curl -fsS -H "Authorization: Bearer $BILLING_API_TOKEN" "$BILLING_API_BASE/customers/$BILLING_CUSTOMER_ID/invoices" | jq . > "$WORKDIR/raw/billing_invoices.json"
else
  printf 'null\n' > "$WORKDIR/raw/billing_customer.json"
  printf '[]\n' > "$WORKDIR/raw/billing_invoices.json"
fi

if [ -n "$SUPPORT_CONTACT_ID" ]; then
  curl -fsS -H "Authorization: Bearer $SUPPORT_API_TOKEN" "$SUPPORT_API_BASE/contacts/$SUPPORT_CONTACT_ID" | jq . > "$WORKDIR/raw/support_contact.json"
  curl -fsS -H "Authorization: Bearer $SUPPORT_API_TOKEN" "$SUPPORT_API_BASE/contacts/$SUPPORT_CONTACT_ID/tickets" | jq . > "$WORKDIR/raw/support_tickets.json"
else
  printf 'null\n' > "$WORKDIR/raw/support_contact.json"
  printf '[]\n' > "$WORKDIR/raw/support_tickets.json"
fi

(
  cd "$WORKDIR"
  find raw -type f -print0 | sort -z | xargs -0 sha256sum > manifest.sha256
)

jq -n \
  --arg generated_at "$TS" \
  --arg email "$EMAIL" \
  --slurpfile identity "$IDENTITY_JSON" \
  '{generated_at:$generated_at, subject_input:{email:$email}, identifiers:$identity[0], files:(inputs)}' \
  "$WORKDIR/manifest.sha256" > "$WORKDIR/manifest.json"

ARCHIVE="$ROOT/out/dsar_${TS}_${SAFE_EMAIL}.zip"
(
  cd "$WORKDIR"
  zip -qr "$ARCHIVE" raw manifest.sha256 manifest.json
)

sha256sum "$ARCHIVE" > "$ARCHIVE.sha256"
aws s3 cp "$ARCHIVE" "s3://$S3_BUCKET/$(basename "$ARCHIVE")" --region "$AWS_REGION" --only-show-errors
aws s3 cp "$ARCHIVE.sha256" "s3://$S3_BUCKET/$(basename "$ARCHIVE.sha256")" --region "$AWS_REGION" --only-show-errors

echo "archive=$ARCHIVE"
echo "sha256=$(cut -d' ' -f1 "$ARCHIVE.sha256")"
BASH
chmod +x bin/export-dsar.sh

You should see no output, and bin/export-dsar.sh should be executable.

Step 4: Run the export for one subject

cd ~/dsar-export
./bin/export-dsar.sh alice@example.com

You should see output shaped like this:

archive=/home/you/dsar-export/out/dsar_20261030T142233Z_alice_example.com.zip
sha256=8b8b6f0d6f8d7b4c4d2e6f8a1d9d0c0f8d8a7e6b5c4d3f2a1b0c9d8e7f6a5b4

Step 5: Generate a time-limited download URL

If you use S3-compatible storage with presigned URLs:

ARCHIVE=$(ls -1t ~/dsar-export/out/dsar_*_alice_example.com.zip | head -1)
aws s3 presign "s3://private-dsar-exports/$(basename "$ARCHIVE")" --expires-in 604800 --region eu-west-1

You should get one HTTPS URL on stdout. 604800 is 7 days.

⚠️ The next step deletes local working files. Do it only after you have verified the uploaded archive and recorded the checksum in your DSAR case record.

Step 6: Remove local raw data after upload

find ~/dsar-export/out -maxdepth 1 -type d -name '*_alice_example.com' -exec rm -rf {} +

You should see no output, and find ~/dsar-export/out -maxdepth 1 -type d should no longer list the subject work directory.

Verify it works

Run all of these checks.

ARCHIVE=$(ls -1t ~/dsar-export/out/dsar_*_alice_example.com.zip | head -1)
sha256sum -c "$ARCHIVE.sha256"
unzip -l "$ARCHIVE"
unzip -p "$ARCHIVE" manifest.json | jq .
unzip -p "$ARCHIVE" raw/identity.json | jq .

Expected results:

• sha256sum -c prints OK
• unzip -l lists at least raw/identity.json, raw/app_user.json, manifest.sha256, and manifest.json
• manifest.json contains the subject input, resolved identifiers, and generated timestamp
• identity.json shows the same user_id used by the SQL exports and the external service IDs used by the API calls

If an API endpoint is misconfigured or your token is wrong, curl -fsS will exit non-zero and print output like this:

curl: (22) The requested URL returned error: 401

If your database query fails, psql exits non-zero and prints output like this:

psql:sql/identity.sql:9: ERROR:  relation "billing_customers" does not exist

[Operator] ---- 1. run export ----> [Export Script]
[Export Script] ---- 2. SQL by user_id ----> [App DB]
[Export Script] ---- 3. API by mapped IDs ----> [Billing API]
[Export Script] ---- 4. API by mapped IDs ----> [Support API]
[Export Script] ---- 5. zip + sha256 ----> [Archive]
[Export Script] ---- 6. upload private ----> [Object Storage]

Common pitfalls

Identity mismatch across services

Mistake: using email everywhere when billing or support actually keys on a separate customer/contact ID. Symptom: local DB files are populated, but billing_customer.json or support_contact.json is null and related arrays are empty. Fix: add the cross-reference joins in sql/identity.sql and export by those mapped IDs, not by email.

psql -tA returns blank because the subject is missing

Mistake: assuming every request email exists in the primary user table. Symptom: script exits with identity lookup returned no data for alice@example.com and exit code 66. Fix: run the identity query manually and confirm the subject exists or switch the canonical input to internal user ID for that case.

API requests fail on redirects or wrong base path

Mistake: pointing BILLING_API_BASE or SUPPORT_API_BASE at a UI host or missing /api/v1. Symptom: curl exits with 22, or curl -I shows a redirect to login.

curl -I https://support.example.internal/contacts/123
HTTP/2 302
location: /login

Fix: use the actual API base URL and test one authenticated endpoint with curl -fsS -H "Authorization: Bearer ..." before running the export.

JSON files contain escaped strings instead of objects

Mistake: selecting row_to_json(...)::text or wrapping JSON twice in SQL. Symptom: jq . raw/app_user.json prints a quoted string or fails with parse error. Fix: return real JSON from SQL, for example SELECT coalesce(json_agg(t), '[]'::json) FROM (...) t.

Archive uploaded but inaccessible to the reviewer

Mistake: uploading to private storage without generating a presigned URL or without recording the checksum. Symptom: the object exists, but the reviewer gets AccessDenied or cannot verify integrity. Fix: run aws s3 presign ... --expires-in 604800 and store both the URL and the SHA-256 value in the case record.

Raw data left on disk after completion

Mistake: keeping the unarchived raw/ directory on a shared build runner or laptop. Symptom: the export is complete, but personal data remains in ~/dsar-export/out/<timestamp>_*. Fix: delete the subject work directory with the find ... -exec rm -rf command after verifying the uploaded archive.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.