Comparing candidates

A comparison doesn't decide what "better" means. It lines up two runs of the same suite. Your promotion rule decides whether the candidate is worth adopting.

An eval run scores one target. Deciding whether to switch to something new (a prompt, a model, a retrieval change, a fine-tuned artifact) takes two runs of the same suite, a comparison, and a rule you'd defend:

import { compareEvaluations, evalSuite, evaluate, grader } from "june/train";

declare function current(input: string): Promise<string>;
declare function proposed(input: string): Promise<string>;

const suite = evalSuite<string, string, string>({
  name: "support",
  cases: [
    { id: "cancel", input: "How do I cancel?", expected: "Billing" },
    { id: "thanks", input: "Thanks, that worked!" },
  ],
  graders: [
    grader("names-the-page", ({ output, expected }) =>
      expected === undefined ? undefined : Number(output.includes(expected)),
    ),
    grader("brief", ({ output }) => (output.split(/\s+/).length <= 40 ? 1 : 0)),
  ],
});

const baseline = await evaluate(current, suite, { name: "current" });
const candidate = await evaluate(proposed, suite, { name: "proposed" });
const comparison = compareEvaluations(baseline, candidate);

// No case may get worse on the grader that must never regress, no more cases
// may fail than before, and the metric you're trying to move must improve by
// 0.05. Deltas only cover graders scored in both runs, so a case the candidate
// failed has no delta; the failed count is what catches it.
const promote =
  comparison.cases.every((c) => (c.deltas["names-the-page"] ?? 0) >= 0) &&
  comparison.failed.candidate <= comparison.failed.baseline &&
  (comparison.scores.brief?.delta ?? 0) >= 0.05;

compare.ts runs a comparison like this on two suites with no API keys, and is checked by June's tests.

What a candidate is

A candidate is anything evaluate() can run: an AI SDK model, a tuned provider model, a locally served adapter, or a function that calls an agent, a pipeline or a June workflow. The baseline is usually whatever runs today.

Only the target should differ between the two runs. Use the same suite, the same graders and, for model graders, the same judge model. If any of those change, both targets have to be evaluated again.

How runs line up

compareEvaluations(baseline, candidate) matches cases by id. Ids are the case's own id, or a hash of its input and expected, so they're the same on every run of the same cases. A case only one run has is listed in missing and left out of every delta.

June doesn't check that the two runs are of the same suite; it only matches ids. Check that baseline.suite and candidate.suite agree and that missing is empty before trusting a comparison.

Deciding

Decide what "better" means before you look at the numbers, and write it down as a promotion rule. June has no promotion function and doesn't pick winners; it gives you the evidence, and a person decides.

A mean can hide a regression. Averaging graders, or cases, lets a gain on one hide a loss on another: a candidate that gets briefer can quietly get less accurate. So a good rule has two parts:

• No regression on what must never get worse: usually a deterministic grader, case by case, and on the golden benchmark.
• A minimum improvement on the one metric you're trying to move.

Run the rule on the comparison, and record it with the result. Common pieces of a rule:

• no regressed cases on the golden benchmark;
• no grader's delta below zero, or below a tolerance you choose for noisy model graders;
• failed.candidate no higher than failed.baseline;
• a minimum win rate in a pairwise preference.

Small suites move a lot from one case. With 20 cases, one case is 5% of a mean. Look at the cases that moved, not just the numbers.

The Copy Agent works this way: copy-agent eval compares a candidate with the active agent on its evidence and its golden benchmark, and only copy-agent promote, run by a person, changes which one answers.

Golden benchmarks and evolving evidence

Two kinds of cases answer different questions, so keep them in separate suites and never average them together:

• A golden benchmark is a fixed, reviewed set of cases that covers everything the app must keep doing well, including what nobody has asked for lately. It changes rarely, on purpose, in its own reviewed change, with a version you bump. A candidate shouldn't regress on it.
• Evidence comes from use: held-out test splits of your data, or eval cases from chat feedback. It follows what people have been asking for, so it shifts as they do, and it's what tells you a candidate fixes real complaints.

Evidence alone lets a candidate get better at this month's requests and quietly worse at everything else. A benchmark alone never learns what people actually need. Run both, report them side by side, and gate on each.

When cases fall into groups (writing modes, topics, languages), add a group to each case's metadata and report the comparison per group as well, so a regression in a small group can't hide in the overall means.

The Copy Agent is a working example. Its golden benchmark is a versioned file of cases by writing mode, and evaluate.tsx runs the active agent and a candidate on the evidence suite and the benchmark separately, compares each, asks a judge for a pairwise preference in both orders, and breaks the benchmark down by category.

Reading a comparison

Means and deltas can disagree, and that's deliberate:

• A run's mean covers every case its grader scored in that run. A case that one run skipped or failed counts toward the other run's mean.
• delta only covers cases the grader scored in both runs, so a case that started failing can't make a grader look better or worse. paired says how many that was. When no case was scored in both runs, delta is missing.
• Means only cover cases that scored, so always read failed next to them. A candidate whose target errors on its hardest cases can show a higher mean.

A case is in regressed when the mean of its graders' changes is below zero, or when it had no errors in the baseline and has some in the candidate. It's in improved when that mean is above zero, or when its errors went away. A change in errors decides first: a case that started failing is only in regressed, and one whose errors went away is only in improved, whatever its scores did. A case with no change, or whose graders cancel out, is in neither.

Averaging a case's graders treats them all as equally important, so a gain on one grader can hide a loss on another. Check per-grader deltas too, or compute your own rule from cases, which has every change. In compare.ts, the candidate gets briefer and less polite on the same cases, so no case is regressed while the polite grader's delta is negative.

Wins and pairwise preference

Pairwise preference is an optional second method, for qualities graders can't score well on their own. compareEvaluations() compares scores each target earned on its own. A judge model asked to choose between two replies to the same case is often more sensitive than two absolute scores, and gives a win rate.

That's an ordinary eval: its cases are pairs of outputs from the two runs, its target makes the choice, and its grader scores the choice:

import { openai } from "june/sdk/openai";
import { generateText, jsonSchema, Output } from "ai";
import { evalSuite, evaluate, grader, type EvalRun } from "june/train";

declare const baseline: EvalRun<string, string, string>;
declare const candidate: EvalRun<string, string, string>;

interface Pair {
  input: string;
  baseline: string;
  candidate: string;
}

const replies = new Map(candidate.cases.map((c) => [c.id, c.output]));
const pairs = baseline.cases.flatMap((left) => {
  const right = replies.get(left.id);
  return left.output !== undefined && right !== undefined
    ? [
        {
          id: left.id,
          input: { input: left.input, baseline: left.output, candidate: right },
        },
      ]
    : [];
});

async function choose(pair: Pair, [a, b]: string[], signal: AbortSignal) {
  const { output } = await generateText({
    model: openai("gpt-5-mini"),
    output: Output.object({
      schema: jsonSchema<{ winner: "A" | "B" | "tie" }>({
        type: "object",
        properties: { winner: { type: "string", enum: ["A", "B", "tie"] } },
        required: ["winner"],
        additionalProperties: false,
      }),
    }),
    prompt: `Request: ${pair.input}\n\nA: ${a}\n\nB: ${b}\n\nWhich reply is better?`,
    abortSignal: signal,
  });
  return output.winner;
}

// Ask in both orders, and call it a tie when the judge disagrees with itself.
const judge = async (pair: Pair, { signal }: { signal: AbortSignal }) => {
  const [first, second] = await Promise.all([
    choose(pair, [pair.baseline, pair.candidate], signal),
    choose(pair, [pair.candidate, pair.baseline], signal),
  ]);
  if (first === "B" && second === "A") return "candidate";
  if (first === "A" && second === "B") return "baseline";
  return "tie";
};

const preferences = await evaluate(
  judge,
  evalSuite({
    name: "support preference",
    cases: pairs,
    graders: [
      grader<Pair, string>("preference", ({ output }) => {
        if (output === "candidate") return 1;
        if (output === "baseline") return 0;
        return 0.5;
      }),
    ],
  }),
  { name: "gpt-5-mini" },
);

preferences.scores.preference?.mean; // the candidate's win rate, ties counted as half

Keeping the original case ids lets you line each choice up with the comparison's per-case changes. A judge is a model, so record which one it was, and evaluate both candidates again when it changes.

Recording what was compared

EvalRuns and EvalComparisons are plain JSON. June doesn't store them; save them wherever suits you, next to anything you promote. Each run already holds every case's input, output, scores, grader details, errors and skipped graders, its target's name and when it ran.

A comparison is only meaningful with what it compared, so record with it:

• the suite's name and a fingerprint of its cases, such as a SHA-256 of their JSON, and the benchmark's version;
• each grader's name, what it measures, and whether it's deterministic or judged by a model;
• the judge model, for model graders and preferences;
• what each target was: a model id, a prompt version, or a fine-tune's manifest.

Then a later reader can tell whether two comparisons measured the same thing, and a candidate evaluated on an older benchmark can be evaluated again before it's promoted.

Comparing a fine-tune

A fine-tune is one more candidate. Write the suite and run the baseline before training, on a test split the trainer never sees. After training, serve the artifact so it can be a target (see Artifacts), evaluate it on the same suite, and compare. If the baseline already does well, or a prompt change gets the same gain, you may not need to fine-tune at all.

In a workflow, <Evaluate> under a <FineTune> re-runs when the fine-tune trains something new, and a function child can compare the two runs.