Skip to content

Judges

A Judge is an LLM that scores text against a rubric with structured output.

from simsim import Judge
judge = Judge(make_llm("openai:gpt-4o", temperature=0),
              rubric="Rate how cooperative this negotiation was.\n\n{text}",
              schema={"reasoning": (str, "brief justification, before the score"),
                      "cooperation": (int, 1, 10)})
judge.score(transcript_text)            # {"reasoning": ..., "cooperation": 7}

The rubric can use {text} and any keyword you pass to score(text, **context); without {text}, the text is appended. Put a reasoning field first so the judge explains before it scores. Malformed answers are repaired like any structured output; if repair fails the result is {"error": ...} instead of an exception.

Three ways to use one

Simulation(env, agents, evaluators={"judge": judge.evaluator()})        # score every run
Conversation(..., judge=judge.stopper("Have they agreed on a price?"))   # end runs when it says yes
from simsim import rescore
rescore("runs/exp/results.jsonl", new_judge, render=lambda row: ..., out="rescored.jsonl")

evaluator(render=...) scores render(result) (default: the transcript). rescore scores finished runs again, for example with a stronger judge or a new rubric, without re-running any simulation.

Good practice

  • Use a different, ideally stronger, model family as the judge than as the agents, and report it.
  • Validate the judge on a sample against human ratings before relying on it.
  • Keep the rubric fixed across conditions; it is part of the measurement.
  • Prefer probes and surveys of the agents themselves, or programmatic checks of the world (score=), where they can answer the question.