Collect a gold-standard example set to measure your AI feature

TL;DR — To measure an AI feature well, you need a small, clean set of real examples with the correct expected answer beside each one. The fastest path is to export 50-200 real cases, remove sensitive data, label the right outcome in a spreadsheet, and save the file as CSV with consistent columns. Reading time: ~5 min

Goal

When you finish, you will have a single review-ready file (usually CSV or spreadsheet) containing representative real-world examples, the correct expected result for each example, and enough notes to use the file as the baseline your AI feature is measured against.

Prerequisites

• Access to the system where the AI feature will be used, with permission to view historical examples (for example: support tickets, emails, chat transcripts, documents, forms, or product records)
• Permission to export data from that system, or access to the admin dashboard where exports can be created
• A spreadsheet tool: Google Sheets, Microsoft Excel, or LibreOffice Calc
• A place to store the file securely: your company drive, SharePoint, Google Drive, or another approved internal storage location
• 50-200 real examples to start with; if volume is low, collect at least 30
• A person who knows the correct outcomes for edge cases (unusual cases), even if that person is you
• A list of any sensitive fields to remove before sharing: names, emails, phone numbers, account IDs, addresses, payment details, health data, or internal secrets
• If you will save as CSV, confirm your spreadsheet app can export CSV UTF-8 (text encoding for special characters)

Steps

Step 1: Decide the exact task you are measuring

Write one sentence in a document or at the top of your spreadsheet using this template:

Task: Given <input>, the AI should produce <output>.

Example:

Task: Given a customer support email, the AI should assign the correct category.

What you should see when this succeeds: one clear sentence that describes one task only, not multiple tasks combined.

Step 2: Create the evaluation sheet with fixed columns

In your spreadsheet tool, create a sheet named evaluation_examples with this header row exactly:

example_id,source,input,expected_output,accepted_alternatives,notes,difficulty,contains_sensitive_data,reviewed_by,status

If you prefer to paste headers into the first row, paste this line:

example_id	source	input	expected_output	accepted_alternatives	notes	difficulty	contains_sensitive_data	reviewed_by	status

What you should see when this succeeds: the first row contains these 10 column names exactly, one per column.

Step 3: Export real examples from your current system

Use the dashboard export path in the system that already contains real cases. In your provider's dashboard, look for a path similar to one of these:

Admin/Settings → Export
Reports → Export data
Data → Records → Export CSV
Inbox/Conversations → Export
Tickets/Cases → Export

Export only the fields you need for the task. Use this minimum field list:

record ID, created date, input text or source content, current human-made outcome or label

If your system has no export button, ask your agency for a one-time export using this exact request:

Please export 100 recent real examples for AI evaluation with these fields: record ID, created date, raw input text, and the final human-approved outcome. Save as CSV UTF-8.

What you should see when this succeeds: you have a CSV or spreadsheet file with real historical examples and their human outcome.

Step 4: Copy a representative sample into the evaluation sheet

From the export, copy 50-200 rows into evaluation_examples. Include a mix of easy, normal, and difficult cases. Use these literal values in the difficulty column:

easy
medium
hard

Use this target mix if you have enough examples:

Fill the columns like this:

• example_id: the original record ID, or ex-001, ex-002, ex-003 if needed
• source: where it came from, such as support_email, chat, form_submission
• input: the exact text or content the AI would receive
• expected_output: the correct answer a human would want
• accepted_alternatives: other answers you would also count as correct, separated by |
• notes: short explanation for unusual cases
• difficulty: easy, medium, or hard
• contains_sensitive_data: yes or no
• reviewed_by: reviewer name or initials
• status: start with needs_review

What you should see when this succeeds: your sheet has a balanced sample, not just the newest or easiest records.

Step 5: Remove or mask sensitive data before sharing

⚠️ If you share raw customer data with a vendor or agency without approval, you may create a privacy or compliance issue. Do this step before sending the file outside your organization.

In the spreadsheet, replace sensitive values inside input with placeholders like these:

[NAME]
[EMAIL]
[PHONE]
[ACCOUNT_ID]
[ADDRESS]

Also set contains_sensitive_data to no after masking is complete.

Use your spreadsheet's Find and Replace menu path, typically:

Edit → Find and replace

What you should see when this succeeds: the examples still make sense, but direct personal or secret data is replaced with placeholders.

Step 6: Review and correct the expected outputs

Read each row and update expected_output to the answer you would want the AI to produce today. Do not copy known bad historical labels forward just because they already exist.

Set status to one of these exact values:

approved
needs_review
rejected

Use approved only when a human reviewer agrees the row is correct and usable.

What you should see when this succeeds: every row has a clear expected result, and usable rows are marked approved.

Step 7: Add edge cases that matter to your business

Add 5-20 rows manually for cases that are rare but important. Examples:

• very short inputs
• very long inputs
• ambiguous wording
• misspellings
• multiple intents in one message
• empty or nearly empty input
• cases that previously caused complaints

Create these rows directly in the spreadsheet and set source to:

manual_edge_case

What you should see when this succeeds: the sheet includes not only common cases, but also the failures you most want to catch.

Step 8: Save and export the final file

Rename the file using this pattern:

ai-eval-examples-YYYY-MM-DD

Then export as CSV UTF-8 using your spreadsheet menu, typically:

File → Download/Export → CSV UTF-8 (.csv)

If your agency asked for XLSX instead, use:

File → Download/Export → Microsoft Excel (.xlsx)

What you should see when this succeeds: you have a final file with stable columns and approved examples ready to send.

Verify it works

Run these checks before you send the file:

1. Open the final file and confirm the header row is exactly:

example_id,source,input,expected_output,accepted_alternatives,notes,difficulty,contains_sensitive_data,reviewed_by,status

1. Filter the status column and confirm the rows you plan to use are all:

approved

1. Filter contains_sensitive_data and confirm the count for yes is:

0

1. Spot-check 10 random rows and confirm each one has:

- a real input
- a clear expected output
- difficulty filled in
- reviewer filled in

1. Re-open the exported CSV in your spreadsheet app and confirm special characters still look correct. You should see normal punctuation and accented characters, not broken symbols.

Common pitfalls

Using only easy or recent examples

Mistake: exporting the latest 50 records and stopping there.

Symptom: the AI looks great in testing but fails on messy real-world cases.

Fix: add a deliberate mix of easy, medium, and hard, plus 5-20 manual edge cases.

Keeping bad historical labels as the "truth"

Mistake: treating old human labels as automatically correct.

Symptom: the AI is marked wrong when it gives the answer you actually want now.

Fix: review each expected_output and update it to today's correct answer before marking approved.

Sharing sensitive data by accident

Mistake: exporting raw customer text and sending it without masking names, emails, or IDs.

Symptom: privacy review delays the project or blocks the file from being used.

Fix: run Edit → Find and replace, replace sensitive values with placeholders, and confirm contains_sensitive_data = no for all shared rows.

Mixing multiple tasks in one sheet

Mistake: combining classification, summarization, and extraction examples in the same evaluation file.

Symptom: reviewers disagree on what counts as correct, and scores are meaningless.

Fix: use one sheet and one task sentence per evaluation set; create separate files for separate tasks.

Inconsistent expected outputs

Mistake: using different wording for the same correct answer, such as Billing, billing issue, and invoice.

Symptom: the AI appears inconsistent to score, even when it is effectively correct.

Fix: standardize the allowed answers in expected_output and put acceptable variants in accepted_alternatives separated by |.

Exporting in the wrong file format or encoding

Mistake: sending a proprietary format or a CSV with broken character encoding.

Symptom: rows import badly, columns shift, or special characters appear corrupted.

Fix: export with File → Download/Export → CSV UTF-8 (.csv) and re-open the file once to confirm it looks normal.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.