Preserve forensic evidence from a cloud VM before rebuilding it

TL;DR — Before you rebuild a cloud workload, capture three things in this order: volatile state from the running instance, provider-side evidence such as disk snapshots and instance metadata, and off-host logs/config backups. The most common failure is stopping or deleting the instance too early; take a crash-consistent disk snapshot first, then export memory/process/network evidence, then hash and store everything in write-restricted object storage. Reading time: ~5 min

Goal

When you finish, you will have a timestamped evidence bundle containing instance metadata, process/network state, system logs, configuration, package inventory, and one or more disk snapshots, all hashed and copied to durable storage so you can rebuild the workload without destroying what you need for later analysis.

Prerequisites

• SSH or console access to the workload as root or a user with sudo — check with sudo -v
• Access to your cloud provider account with permission to read instance metadata and create disk snapshots/images
• bash, tar, gzip, sha256sum, journalctl, ss, ps, ip, find, df installed on the instance
• aws CLI v2, gcloud, or az CLI if you will script snapshot creation from your workstation — check with aws --version, gcloud version, or az version
• Object storage or another evidence destination with restricted write access, such as an S3 bucket or equivalent
• At hand: instance ID, region/zone, attached volume IDs, incident ticket number, and the UTC time window you care about

Steps

Step 1: Create a case directory and record the clock

CASE_ID="IR-2026-001"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
HOST="$(hostname -s)"
OUT="/root/evidence-${CASE_ID}-${HOST}-${TS}"
mkdir -p "$OUT"
printf 'case_id=%s\nutc_now=%s\nhost=%s\n' "$CASE_ID" "$TS" "$HOST" | tee "$OUT/00-case.txt"
date -u '+%Y-%m-%dT%H:%M:%SZ' | tee "$OUT/00-utc-now.txt"
uptime | tee "$OUT/00-uptime.txt"

Success looks like files under /root/evidence-... and UTC timestamps printed to stdout.

⚠️ Do not stop, reboot, resize, detach disks, rotate logs, or terminate the instance before Step 3. Those actions destroy volatile evidence and can alter timestamps.

Step 2: Capture volatile state from the running instance

uname -a > "$OUT/10-uname.txt"
id > "$OUT/10-id.txt"
who -a > "$OUT/10-who.txt"
last -F -n 50 > "$OUT/10-last.txt" || true
ps auxwwf > "$OUT/11-ps-auxwwf.txt"
ss -pant > "$OUT/12-ss-pant.txt"
ss -plant > "$OUT/12-ss-plant.txt"
ip addr show > "$OUT/13-ip-addr.txt"
ip route show table all > "$OUT/13-ip-route.txt"
arp -an > "$OUT/13-arp.txt" || ip neigh show > "$OUT/13-ip-neigh.txt"
lsmod > "$OUT/14-lsmod.txt"
mount > "$OUT/15-mount.txt"
df -hT > "$OUT/15-df-hT.txt"
crontab -l > "$OUT/16-crontab-root.txt" 2>&1 || true
systemctl list-units --type=service --all > "$OUT/17-systemd-services.txt"
systemctl list-timers --all > "$OUT/17-systemd-timers.txt"
journalctl --utc --no-pager -S '-24 hours' > "$OUT/18-journal-last-24h.txt"

Success looks like multiple 10-* through 18-* files with non-zero sizes in the case directory.

Step 3: Snapshot every attached disk from the provider side

⚠️ Snapshots can be crash-consistent, not application-consistent. If the workload is still serving traffic and you can tolerate a brief pause, freeze the filesystem first; otherwise snapshot immediately and note that fact in the case file.

On the instance, identify block devices:

lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,SERIAL,UUID > "$OUT/20-lsblk.txt"
cat "$OUT/20-lsblk.txt"

Success looks like a table listing root and data volumes.

From your workstation or cloud shell, create snapshots for each attached volume. Use the command set for your provider.

AWS:

INSTANCE_ID="i-0123456789abcdef0"
REGION="us-east-1"
aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --region "$REGION" --query 'Reservations[0].Instances[0].BlockDeviceMappings[].Ebs.VolumeId' --output text
aws ec2 create-snapshot --region "$REGION" --volume-id vol-aaaaaaaaaaaaaaaaa --description "${CASE_ID} ${TS} ${HOST} root"
aws ec2 create-snapshot --region "$REGION" --volume-id vol-bbbbbbbbbbbbbbbbb --description "${CASE_ID} ${TS} ${HOST} data"

GCP:

ZONE="us-central1-a"
DISK1="instance-root-disk"
gcloud compute disks snapshot "$DISK1" --zone "$ZONE" --snapshot-names "${CASE_ID,,}-${HOST,,}-${TS,,}-root"

Azure:

RESOURCE_GROUP="rg-prod"
DISK_ID="/subscriptions/.../resourceGroups/rg-prod/providers/Microsoft.Compute/disks/vm01_OsDisk_1_..."
az snapshot create --resource-group "$RESOURCE_GROUP" --name "${CASE_ID}-${HOST}-${TS}-os" --source "$DISK_ID"

Success looks like a snapshot ID or operation name returned by the CLI.

Step 4: Export instance metadata and cloud control-plane details

Run the metadata command that matches your provider.

AWS IMDSv2:

TOKEN=$(curl -sS -X PUT 'http://169.254.169.254/latest/api/token' -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600')
curl -sS -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document | tee "$OUT/30-instance-identity.json"
curl -sS -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/info | tee "$OUT/30-iam-info.json"

GCP:

curl -sS -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/?recursive=true' | tee "$OUT/30-gcp-metadata.txt"

Azure:

curl -sS -H Metadata:true 'http://169.254.169.254/metadata/instance?api-version=2021-02-01' | tee "$OUT/30-azure-metadata.json"

Also capture cloud-side instance description from your workstation:

# Example: AWS
aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --region "$REGION" > "${CASE_ID}-${HOST}-${TS}-describe-instances.json"

Success looks like JSON or text metadata files with instance IDs, image IDs, and network details.

Step 5: Collect logs, configs, and package inventory

tar --xattrs --acls --selinux -czf "$OUT/40-etc.tgz" /etc
journalctl --utc --no-pager -S '-7 days' > "$OUT/41-journal-last-7d.txt"
find /var/log -type f -maxdepth 3 -print0 | tar --null -czf "$OUT/42-var-log.tgz" --files-from -
if command -v dpkg >/dev/null 2>&1; then dpkg-query -W -f='${Package}\t${Version}\n' > "$OUT/43-packages.tsv"; fi
if command -v rpm >/dev/null 2>&1; then rpm -qa > "$OUT/43-packages.txt"; fi
if command -v docker >/dev/null 2>&1; then docker ps -a --no-trunc > "$OUT/44-docker-ps.txt"; docker images --digests > "$OUT/44-docker-images.txt"; fi
if command -v kubectl >/dev/null 2>&1; then kubectl get pods -A -o wide > "$OUT/45-k8s-pods.txt"; fi

Success looks like tarballs and package lists created without Permission denied errors.

Step 6: Hash the evidence and copy it to durable storage

cd /root
FINAL_TAR="${OUT}.tgz"
tar -czf "$FINAL_TAR" "$(basename "$OUT")"
sha256sum "$FINAL_TAR" | tee "${FINAL_TAR}.sha256"

Success looks like a .tgz and matching .sha256 file.

Copy to object storage. Example with AWS S3:

BUCKET="s3://my-ir-evidence-bucket/${CASE_ID}/"
aws s3 cp "$FINAL_TAR" "$BUCKET"
aws s3 cp "${FINAL_TAR}.sha256" "$BUCKET"

Success looks like upload: lines for both files.

Step 7: Rebuild only after evidence is off-host

sha256sum -c "${FINAL_TAR}.sha256"

Success looks like:

/root/evidence-IR-2026-001-vm01-20261030T142233Z.tgz: OK

At this point you can terminate and rebuild the workload using your normal image/infra process.

Verify it works

Run these checks before you rebuild:

ls -lh /root/evidence-*.tgz /root/evidence-*.sha256
sha256sum -c /root/evidence-*.sha256

Expected output shape:

-rw-r--r-- 1 root root 184M Oct 30 14:25 /root/evidence-IR-2026-001-vm01-20261030T142233Z.tgz
-rw-r--r-- 1 root root  117 Oct 30 14:25 /root/evidence-IR-2026-001-vm01-20261030T142233Z.tgz.sha256
/root/evidence-IR-2026-001-vm01-20261030T142233Z.tgz: OK

Verify snapshots exist in the provider:

# AWS example
aws ec2 describe-snapshots --owner-ids self --filters "Name=description,Values=${CASE_ID}*" --query 'Snapshots[].{Id:SnapshotId,State:State,StartTime:StartTime}' --output table

Expected state is pending or completed; do not rebuild until snapshot creation has at least started and returned an ID.

Common pitfalls

Deleting or stopping the instance before collecting volatile state

Mistake: terminating, rebooting, or auto-healing the workload first. Symptom: ps, ss, who, and recent in-memory-only indicators are gone; only disk artifacts remain. Fix: run Step 2 first, then create snapshots, then rebuild.

Taking only an in-guest backup and no provider snapshot

Mistake: copying /etc and /var/log but skipping block-level snapshots. Symptom: later you cannot recover deleted files, unallocated space artifacts, or exact disk state. Fix: create snapshots for every attached volume in Step 3, even if you also tar logs and configs.

Metadata service requests fail with 401 or empty output

Mistake: using AWS IMDSv1 on an instance that requires IMDSv2, or missing provider-specific headers. Symptom:

curl: (22) The requested URL returned error: 401

Fix: use the exact metadata commands in Step 4, including the IMDSv2 token or Metadata-Flavor/Metadata:true header.

Evidence archive is created on a nearly full root disk

Mistake: writing a large tarball to /root when the filesystem is already close to 100%. Symptom:

tar: write error: No space left on device

Fix: check df -hT in Step 2; if root is tight, write OUT and the final tarball to a mounted data disk or stream directly to object storage.

Hash file does not match after upload

Mistake: uploading a partially written archive or recompressing it during transfer tooling. Symptom:

sha256sum: WARNING: 1 computed checksum did NOT match

Fix: run sha256sum -c locally before upload, then download the object and verify the same checksum again.

Snapshot exists but misses the data volume

Mistake: snapshotting only the OS disk and forgetting attached application or database volumes. Symptom: the rebuilt investigation mount shows / but not /var/lib/postgresql, /var/lib/docker, or your app data. Fix: use lsblk and provider block-device mapping output together; create a snapshot for every attached volume ID, not just the boot disk.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.