A small, honest eval harness beats vibes

I have lost count of how many times a prompt change "felt better" in a demo and then quietly broke three things in production. The fix was not a fancy platform. It was forty lines of Python that I run before every release.

If you are building anything on top of a language model, you need a way to answer one question with a number: did this change make the output better or worse? Without that, you are tuning prompts by vibes, and vibes do not survive contact with real traffic.

Start with cases, not metrics

The hardest part is not the code. It is writing down what "correct" means. Collect ten to thirty real inputs, ideally pulled from actual usage, and write the expected behavior for each. You do not need perfect golden answers. You need a check that is cheap to run and hard to game.

A few kinds of checks that pay for themselves:

• Exact or fuzzy match against a known answer for factual tasks.
• A required substring or JSON shape for structured output.
• A rule the answer must never break, like leaking a system instruction.
• A second model grading the answer on a one to five scale, used sparingly.

Keep the first version dumb. You can add a model grader later, but string and schema checks catch most regressions for free, and they never flake.

The harness

Here is the shape I keep reaching for. It reads cases from a file, runs them, and prints a pass rate you can compare across versions.

import json, statistics

def run_eval(cases, generate):
    results = []
    for c in cases:
        out = generate(c["input"])
        passed = c["must_include"] in out
        results.append({"id": c["id"], "passed": passed, "output": out})
    rate = statistics.mean(1 if r["passed"] else 0 for r in results)
    print(f"pass rate: {rate:.0%}  ({sum(r['passed'] for r in results)}/{len(results)})")
    return results

with open("cases.json") as f:
    cases = json.load(f)

run_eval(cases, my_model_call)

That is the whole idea. Swap my_model_call for your real function, point it at a JSON file of cases, and you have a regression test for behavior instead of code.

A test you run every release is worth more than a benchmark you read about once.

Wire it into the loop

The harness only helps if it runs without you thinking about it. I commit the cases file next to the code, run the script in CI, and fail the build if the pass rate drops below an agreed line. When the number moves, I want to know on the pull request, not from a support ticket.

Two habits make this stick. First, every time a bug slips through, add it as a new case before you fix it. Your suite grows in exactly the spots that hurt. Second, store each run so you can diff outputs between versions. A dropped pass rate tells you something broke. The diff tells you what.

None of this is novel. Tools like OpenAI Evals and the lm-evaluation-harness do far more, and you should graduate to them when your needs grow. But the gap that actually costs teams money is the gap between zero evals and one. Closing that gap takes an afternoon.

Write the cases. Run the number. Ship when it goes up.