Not so fast.

An evaluation tells you something about the examples you tested, the criteria you chose, and the evaluator you used. It does not automatically tell you how the system will behave on a new request or an unusual edge case.

I ran into this while building the evaluation pipeline for my AI Waiter Synthetic Data Generator. A small model generates restaurant Q&A, I label it by hand, and an independent LLM judge scores the same items. After iterating on the judge rubric, mean agreement with my labels rose from 70.2% to 96.0% across the same 42 examples. Every change is recorded in the iteration log.

That was real progress. It was also a reminder of what the number could---and could not---prove.

(Code links point to commit 46988db so line numbers stay accurate.)

Define what "good" means, dimension by dimension

For an AI waiter, a response should be consistent with the menu, useful, complete, within the waiter's role, and careful about dietary and food-safety concerns. Those are related but not interchangeable: an answer can be friendly and wrong about an ingredient, or safe and not answer the question.

So I score them separately. The dimensions live in one shared list, quality_dimensions.py, imported by both the labeling tool and the judge, so the two can never drift apart:

QUALITY_DIMENSIONS = [
    "answer_completeness", "safety_specificity", "menu_realism",
    "scope_appropriateness", "context_clarity", "tip_usefulness",
]

A single number is useful for tracking a trend. It is much less useful for explaining what needs fixing.

Build the set around failure modes

What matters is not how many examples you have, but what they cover. The project generates one batch per category defined in config.py: menu guidance, dietary restrictions and allergies, beverages, order customization, and service recovery. Writing that list down also showed me what's missing: out-of-scope requests, ambiguous requests, and cases where the menu doesn't support a confident answer.

Some failures are cheap to catch before any model looks at them. Step 2 runs deterministic checks: a minimum length for the safety note, a blocklist of generic phrases like "please let us know about any allergies", and question deduplication. (One caveat: today it only reports. Later steps still see the items it would reject.)

For every item, you should be able to answer: What behavior am I testing? and What would count as a failure?

Separate regression testing from generalization

The most useful decision in the experiment was keeping the same 42 examples for every judge-prompt version. The judge reads a saved generation file instead of regenerating, and each run writes to a file named after both prompts, so runs sit side by side:

python3 pipeline.py step4 --generator-prompt=v1 --judge-prompt=v3
# -> output/step4_llm_judge_labels_v1_v3.json

Each item has a stable trace_id, and the analysis step joins human and judge labels on it.

That gives me a regression set: did this change break something I already tested? It doesn't give me a held-out set: does this approach work on examples I didn't tune against? Tune long enough on the same 42 items and you start optimizing for those items. I've established comparability across prompt versions, not generalization. That's a next step, not a result.

Treat the evaluator as part of the system

An LLM judge is not an oracle. It's another model following a rubric---and it fails like software. The first time I switched to gpt-oss-20b, it returned {"pass": true} with no rationale, my Pydantic schema required one, and the run died on the first item. Changing the judge model is a system change.

The bigger lesson was about vague criteria. The baseline judge agreed with me only 9.5% of the time on safety specificity. The default prompt mostly said "be strict." By v3, it had explicit evidence for a pass and explicit edge cases:

Pass when the field ties a risk or a precaution to this order:
- A named allergen, ingredient, diet, or irritant.
- Cross-contact, or a shared or dedicated prep area.
- A doneness or serving temperature for the item ordered.
Edge case: a named hazard plus a generic allergy sentence still passes.

Agreement on that dimension reached 90.5%. If a rubric says "specific," define the evidence. If it says "complete," say whether the answer must repeat every detail or just answer the request---that one clarification moved completeness from 78.6% to 97.6%.

When agreement is low, the cause might be the judge, the rubric, or the human labels. Those need different fixes.

Look at failures, not only the average

My first rubric revision raised mean agreement from 70.2% to 87.7%, while answer completeness dropped from 90.5% to 78.6%. The average improved; one dimension got worse.

Pass rates like these help locate problems, but similar rates can hide disagreement on which items pass. That's why agreement is computed item by item:

agreement_total += 1.0 if human_value == llm_value else 0.0

I review at three levels: the overall trend, each dimension, and the individual disagreements. The third is where debugging happens. The per-category breakdown showed Order Customization passing only 6 of 10 items. Reading the four failures (qa_022, qa_027, qa_028, qa_029) showed one pattern---safety notes that never named a dish---which became a targeted generator-prompt change.

Watch for the quiet failure too: no result at all. If the judge errors on an item, Step 4 skips it, and Step 5 scores the gap as zeros---indistinguishable from a judge that failed it everywhere. "No result" should be its own state.

Be careful with human labels

Human labels are not automatically ground truth. One reviewer---me---labeled the 42 examples, so the percentages measure agreement with my labels, not consensus.

There's a bigger limitation. I passed every item on every dimension. When the human column is all passes, agreement is the judge's pass rate. The 96.0% says the judge passes items I consider good. It says nothing about whether it would fail a bad one, because the set contains none.

The tooling nudged that way, too: in the labeling CLI, pressing Enter counts as a pass. Fast for 252 judgments---but the file can't tell a deliberate pass from a skipped decision.

A mature process needs a second reviewer on at least a subset, and known-bad examples the judge is expected to catch.

A practical loop

No platform required, just discipline:

1. Define the behavior and what failure looks like.

2. Build examples around normal cases, edge cases, and failure modes.

3. Establish reference labels and flag ambiguous ones.

4. Run the evaluation and track per-dimension results.

5. Read the disagreements.

6. Make one targeted change with a stated hypothesis.

7. Re-run the regression set, then check held-out examples.

8. Add important new failures to the regression set.

I keep this in iteration-log.md, where every entry has the same fields: Change, Hypothesis, Result, Decision, Next step. Writing the hypothesis before the run makes it much harder to rationalize the result afterward.

What a good score still can't tell you

Offline evaluation can't guarantee production traffic resembles your set, or that every rare but serious failure is covered. A deployed feature also needs monitoring, user feedback, and explicit thresholds for high-risk behavior. A weak restaurant recommendation and a wrong allergen assurance shouldn't carry the same weight.

For AI Waiter, my next steps are a held-out set, a second reviewer, a severity-ranked review of the remaining disagreements, and folding important failures back into the regression set.

Final thought

Evaluation is not a certificate of reliability. It is a feedback loop for engineering better behavior.

My judge's agreement rose from 70.2% to 96.0% on the same 42 examples. That's a meaningful result for that experiment. The next question is whether it holds on new data and under more independent review.

The code, prompts, and full iteration history are in the repository.

About this series

This is the second entry in Building AI Products: Engineering Notes from the Field, a series about the experiments, implementation decisions, failures, and lessons behind building AI-powered products.

• Project: AI Waiter Synthetic Data Generator

• Previous article: My LLM Judge Went From 70% to 96% Agreement With Humans. Here's What I Learned. --- add the published article URL here.

When I started building the evaluation pipeline for my AI Waiter project, I expected the hard part to be generating realistic restaurant conversations.

It wasn’t.

The harder problem was deciding whether the generated answers were actually good.

An AI waiter needs to do more than sound natural. It should answer menu questions accurately, give useful recommendations, handle dietary restrictions carefully, and stay within the scope of what a waiter should know or do.

I needed a way to evaluate those behaviors systematically. That meant using an LLM as a judge.

My first version agreed with my human reference labels only 70.2% of the time.

After three iterations of the judge prompt, agreement reached 96.0% on the same evaluation set.

The interesting part wasn’t the number itself. It was what the process revealed about evaluation design, rubric ambiguity, and the risk of trusting an LLM judge before understanding how it fails.

The setup: an AI waiter needs more than a fluent answer

The project is an AI Waiter Synthetic Data Generator. Its pipeline generates synthetic data for an AI waiter and evaluates the quality of that data across multiple dimensions.

For this experiment, I focused on the judge responsible for assessing answer quality.

I used six evaluation dimensions:

• Safety specificity: Does the answer identify a relevant hazard or precaution rather than making a generic safety claim?

• Tip usefulness: Does the recommendation offer something genuinely useful to the customer?

• Menu realism: Is the answer plausible given the restaurant’s menu?

• Scope appropriateness: Does the answer stay within the waiter’s role?

• Context clarity: Is the relevant context clear enough to understand the answer?

• Answer completeness: Does the answer actually address the customer’s request?

These dimensions cover different failure modes. An answer can be fluent but incomplete, helpful-sounding but unsafe, or perfectly reasonable in isolation but inconsistent with the restaurant’s menu.

A single overall quality score would make those distinctions difficult to diagnose. Separate dimensions gave me a way to see where the judge was failing.

The experiment: a fixed set and a human reference

I used an evaluation set of 42 items. One human reviewer labeled the examples, and those labels served as the reference against which I measured the judge.

I kept the same 42 items for every judge-prompt version. That was important: it meant the measured changes came from changing the judge prompt, rather than from evaluating different examples each time.

The experiment was straightforward:

1. Run the judge against the fixed evaluation set.

2. Compare its decisions with the human reference labels.

3. Inspect the dimensions with the weakest agreement.

4. Revise the rubric to make the pass/fail criteria more explicit.

5. Rerun the same set and compare the results.

One important limitation: these are agreement scores with one reviewer, not a consensus among multiple annotators. The human labels are a useful reference, but they are not necessarily objective ground truth.

The baseline: 70.2% agreement

The first results made the problem obvious. The judge was doing reasonably well on some dimensions and badly on others.

The weakest results were safety specificity and tip usefulness.

That pattern mattered. The judge was already good at identifying some fairly clear-cut problems, such as whether an answer fit the menu or stayed within the waiter’s role. It struggled when the quality dimension required interpretation.

The issue wasn't necessarily that the model couldn't understand the answer. The rubric left too much room for the model to decide what a passing answer should look like.

For example, what counts as a specific safety note? Is mentioning an allergy enough? Does the answer need to name an ingredient? What about cross-contact or a temperature precaution?

If those expectations aren't explicit, the judge has to invent its own interpretation of the rubric.

I decided to make the criteria more concrete before trying anything more elaborate.

Iteration 1: make the rubric testable

For the first revision, I added clearer pass criteria, examples of failures, and edge cases—particularly for safety_specificity and tip_usefulness.

The goal was to move from a subjective instruction to a rubric that could be applied consistently.

Here's an excerpt from the safety rubric in judge prompt v1:

Pass criteria: it names the hazard or precaution for this scenario.

The rubric then gave concrete examples of what would qualify: a named allergen or ingredient to avoid, cross-contact or a shared or dedicated preparation area, a doneness temperature for the item ordered, a named spice or irritant, or a service problem paired with a fix.

It also clarified what should fail. Generic statements such as “Prepared safely” weren't sufficient if they didn't identify a hazard.

That distinction sounds small, but it changes the task. Instead of asking the judge to decide whether a safety note feels specific, it asks the judge to look for identifiable evidence.

The results improved substantially.

Mean agreement increased by 17.5 percentage points, from 70.2% to 87.7%.

Safety specificity rose from 9.5% to 59.5%, while tip usefulness rose from 42.9% to 88.1%.

But there was a regression: answer completeness dropped from 90.5% to 78.6%.

This was a useful reminder that improving one part of a rubric can expose problems—or introduce new ones—in another. I hadn't solved evaluation quality; I'd made progress and uncovered the next issue.

Iteration 2: fix what “complete” actually means

The completeness regression forced me to look more carefully at the distinction between a complete answer and an exhaustive answer.

Those aren't the same thing.

If a customer asks whether a particular dish can be modified, a direct answer about that modification may be complete. The waiter doesn't necessarily need to repeat the entire menu, enumerate every service step, or restate every constraint.

A judge that expects all potentially relevant information to be repeated can mark concise, correct answers as incomplete.

I clarified the criterion in judge prompt v2:

Pass criteria: the answer gives a direct reply (a dish, a choice, a modification, or the next step for this request).

The rubric also explicitly said not to require an answer to repeat every menu item, service step, or constraint.

The change narrowed the evaluation to what the customer actually needed.

Completeness recovered to 97.6%, and mean agreement reached 91.7%.

The broader lesson: a rubric should define the evidence required to pass, not quietly impose an ideal answer format.

This matters especially in conversational AI, where a short answer can be more appropriate than a comprehensive one.

Iteration 3: refine safety without rewarding generic claims

Even after two iterations, safety specificity was the weakest dimension at 64.3%.

The initial rubric required a named hazard or precaution. That helped, but the remaining disagreements suggested that the criteria still didn't capture all the forms a relevant safety note could take.

A useful note might connect a specific order to a precaution, describe a temperature concern, or identify a service problem and the action the waiter should take. A rigid interpretation of “specificity” could miss some of those cases.

In judge prompt v3, I refined the criterion so the safety field could pass when it tied a risk or precaution to the order. The examples included named allergens, ingredients, dietary restrictions or irritants; cross-contact; temperature; and service problems paired with a waiter action.

The failure criterion also became more explicit: the field should fail when it names neither the order nor an action.

This was not simply lowering the bar for safety. The aim was to distinguish a relevant, order-specific safety note from an empty assertion that something is safe.

The results:

Safety specificity improved by 26.2 percentage points in this final iteration, while the other reported dimensions held steady.

From the original baseline to v3, mean agreement increased from 70.2% to 96.0%, and safety specificity increased from 9.5% to 90.5%.

That was the result I was looking for—not just a higher aggregate score, but a rubric that was much more consistent on the dimension that had initially failed most often.

The full progression

Mean agreement with human reference labels

Across the three iterations, the process was less about writing a clever prompt and more about diagnosing ambiguity:

• Baseline → v1: Make the weakest dimensions operational with explicit pass criteria and failure examples.

• v1 → v2: Correct a regression by defining completeness around answering the request, rather than repeating everything.

• v2 → v3: Refine the safety rubric so it recognizes relevant order-specific precautions without accepting generic claims.

The overall score tells one part of the story. The per-dimension scores explain why each change was worth making.

How to reproduce the experiment

The judge prompts are versioned in the repository, so the iterations can be inspected directly:

• Judge prompt v1

• Judge prompt v2

• Judge prompt v3

From the project directory, the evaluation command is:

python3 pipeline.py --generator-prompt=X --judge-prompt=Y

Replace X and Y with the generator and judge prompt versions you want to run.

For a meaningful comparison, keep the evaluation set fixed while changing the judge prompt. That's what I did here: the same 42 items were used for every judge-prompt version, with the same human labels as the reference.

One qualification: reproducing the comparison also depends on keeping the relevant model configuration and other evaluation conditions consistent. The command selects prompt versions, but by itself doesn't establish that every external dependency is identical across runs.

What I learned about LLM-as-a-judge evaluation

1. Aggregate scores can hide the actual problem

The baseline mean agreement of 70.2% didn't tell me which part of the judge needed attention. The dimension-level breakdown did.

Safety specificity was at 9.5%, while context clarity was already at 97.6%. Those two dimensions shouldn't be debugged in the same way.

When evaluating an LLM judge, inspect the confusion patterns and disagreements for individual dimensions before changing the whole prompt.

2. Examples make abstract criteria actionable

Words like specific, useful, and complete sound clear until you have to label borderline cases.

Concrete pass criteria and failure examples reduce the space for inconsistent interpretation. They also make it easier to explain why a particular answer should pass or fail.

The point isn't to make every possible case fit a list of examples. It's to establish the intended boundary and illustrate how to apply it.

3. Prompt changes can cause regressions

Judge v1 improved mean agreement dramatically, but completeness fell by 11.9 percentage points compared with baseline.

Without the per-dimension metrics, that regression could have been hidden by the improved overall score.

This is why I treat evaluation prompts as versioned artifacts and rerun a fixed regression set whenever I change the rubric.

4. Don't optimize only for agreement

A judge can agree with a human reviewer more often without necessarily becoming more correct in an absolute sense.

If the rubric is wrong, agreement may reward the wrong behavior. If the reference labels are inconsistent, a high agreement score can reproduce that inconsistency.

In this experiment, the human labels came from one reviewer. That makes the result useful as a measure of consistency with that reviewer's judgments, but it doesn't tell me how another reviewer would label the same examples.

5. A fixed test set is useful—but only for what it covers

Keeping the 42 items constant made the prompt iterations comparable. However, repeated tuning against the same set risks overfitting the judge prompt to those examples.

The next step should be to evaluate the final prompt on a separate set of examples that wasn't used during iteration. I'd also want more than one human reviewer for at least a subset of the data, so I could examine inter-annotator agreement and identify ambiguous labels.

What 96% does—and doesn't—tell me

The final result is encouraging, but it needs to be interpreted narrowly.

What it tells me:

• The judge's agreement with my human reference labels improved substantially on the 42-item evaluation set.

• The biggest improvements came from clarifying criteria for safety specificity, tip usefulness, and answer completeness.

• The final rubric performed much more consistently on the dimensions that initially caused trouble.

What it doesn't establish:

• That the judge is 96% accurate across all AI waiter conversations.

• That the human reference labels represent universal ground truth.

• That the same agreement will hold on unseen examples, different restaurants, or different models.

• That passing these six dimensions is sufficient to guarantee production safety.

In particular, 90.5% agreement on safety specificity is not a safety certification. It is an agreement metric for one dimension on one fixed set.

For a production system, I'd want a broader evaluation set, explicit high-risk test cases, independent human review, and a way to inspect disagreements—not just the headline percentage.

What's next

I'd focus on three things:

1. Build a held-out evaluation set. Test whether the v3 rubric generalizes beyond the 42 examples used to tune it.

2. Measure human disagreement. Have additional reviewers label a subset and distinguish judge errors from ambiguous reference labels.

3. Analyze the remaining failures. A 96.0% mean still leaves disagreements. I'd want to know exactly which examples fail, whether errors cluster around particular dimensions, and whether those errors have different operational risks.

The broader goal is not to get a judge to produce an impressive number. It's to build an evaluation process that helps me make better engineering decisions about the system being evaluated.

Final thought

My biggest takeaway from this experiment is that an LLM judge is only as useful as the criteria it is asked to apply—and the evidence used to validate those criteria.

I started with a judge that agreed with my human reference labels 70.2% of the time. Three prompt iterations later, agreement reached 96.0% on the same 42 examples.

The progress came from treating evaluation like an engineering problem: identify where it fails, make the criteria explicit, rerun a controlled test, and inspect what got better and what got worse.

The number is a milestone, not the finish line. The next test is whether the rubric holds up on data it has never seen.