#Evaluation: gold questions, regression, don’t vibe-check only

A prompt is product behavior. OpenAI’s current guidance: keep prompts in the repo, review them in the same PR as the feature, and run tests and eval cases on every publish. Looking once in a chat box (“seems fine”—a vibe-check) will not tell you what happens with a different user phrasing, a new model snapshot, or two extra noisy retrieved chunks.

#What counts as an eval

One case = fixed input + a decidable expectation. Expectations can be:

• Exact match (an enum label)
• Schema-valid (JSON parses; enums in the set)
• Must-include / must-not (quoted a source sentence; no secrets)
• Tool trace (did search when it should; never called delete_*)
• A human rubric (1–5 “sendable?”) with written scoring notes—not “vibes”

Trying something with no expectation is exploration, not evaluation.

#Gold questions

Start with 10–30 items that match real traffic, not 200 synthetic paragraphs. Label why each exists:

Store gold questions next to the prompt (for example evals/gold.jsonl). Every change to examples or schema must run this set—that is regression.

#How to run regression

Minimum: a script reads jsonl, calls a pinned model snapshot, scores, exits non-zero on failure. You do not need a platform on day one. LangSmith, Promptfoo, and Dify logs can wrap the loop; the cases themselves should be yours.

Practice:

1. Pin a model snapshot (gpt-4.1-mini-YYYY-MM-DD style) so “today’s default model” does not flip the suite
2. Pin the prompt version (git SHA) in the report
3. Read per-case failures before the average: G03 turning red matters more than a 0.02 score wiggle
4. When you edit the prompt to fix one class of failures, run the full gold set
5. For agents, store the trace: which tools, which arguments—not only the final sentence

After launch, sample real logs into the gold set (redact). If new production failures never enter the set, regression is theater.

#Why vibe-check is not enough

Vibes are fine while exploring. As a release gate, at least: all gold questions green + three human reads of the final reply. Humans judge “shippable.” Scripts judge “the contract did not break.”

#A tiny gold file

{"id":"G01","input":"Charged twice, order 8891","expect":{"label":"billing","order_id":"8891"}}
{"id":"G04","input":"How many refund days?<doc>shipping SLA only</doc>","expect":{"abstain":true}}

Score schema first, then business keys. For “must abstain” cases like G04, use a boolean—not a fuzzy “sounds polite.” Agent cases also record tools_used: G01 must not call delete_order.

Print failing rows, then edit the prompt. Tuning adjectives against a mean score is a vibe-check with a script.

#Relation to the four blocks

Eval items should map to the contract: role (lecturing the wrong audience), task (deliverable present), constraints (forbidden zone crossed), format (parseable). A sentence you cannot test is a sentence you did not write.

When you change models or retrieval, run gold questions before you claim “the new model is smarter.” Smart is not a gate. The contract is.

#Next steps

• Pitfalls
• Practical examples