# Comparing candidates

A comparison doesn't decide what "better" means. It lines up two runs of the same suite. Your promotion rule decides whether the candidate is worth adopting.

An [eval](https://june.dev/docs/training/evals.md) run scores one target. Deciding whether to switch to something new (a prompt, a model, a retrieval change, a fine-tuned artifact) takes two runs of the same suite, a comparison, and a rule you'd defend:

```ts
import { compareEvaluations, evalSuite, evaluate, grader } from "june/train";

declare function current(input: string): Promise<string>;
declare function proposed(input: string): Promise<string>;

const suite = evalSuite<string, string, string>({
  name: "support",
  cases: [
    { id: "cancel", input: "How do I cancel?", expected: "Billing" },
    { id: "thanks", input: "Thanks, that worked!" },
  ],
  graders: [
    grader("names-the-page", ({ output, expected }) =>
      expected === undefined ? undefined : Number(output.includes(expected)),
    ),
    grader("brief", ({ output }) => (output.split(/\s+/).length <= 40 ? 1 : 0)),
  ],
});

const baseline = await evaluate(current, suite, { name: "current" });
const candidate = await evaluate(proposed, suite, { name: "proposed" });
const comparison = compareEvaluations(baseline, candidate);

// No case may get worse on the grader that must never regress, no more cases
// may fail than before, and the metric you're trying to move must improve by
// 0.05. Deltas only cover graders scored in both runs, so a case the candidate
// failed has no delta; the failed count is what catches it.
const promote =
  comparison.cases.every((c) => (c.deltas["names-the-page"] ?? 0) >= 0) &&
  comparison.failed.candidate <= comparison.failed.baseline &&
  (comparison.scores.brief?.delta ?? 0) >= 0.05;
```

[`compare.ts`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/compare.ts) runs a comparison like this on two suites with no API keys, and is checked by June's tests.

## What a candidate is

A candidate is anything `evaluate()` can run: an AI SDK model, a tuned provider model, a locally served adapter, or a function that calls an agent, a pipeline or a June workflow. The baseline is usually whatever runs today.

Only the target should differ between the two runs. Use the same suite, the same graders and, for model graders, the same judge model. If any of those change, both targets have to be evaluated again.

## How runs line up

`compareEvaluations(baseline, candidate)` matches cases by id. Ids are the case's own `id`, or a hash of its `input` and `expected`, so they're the same on every run of the same cases. A case only one run has is listed in `missing` and left out of every delta.

June doesn't check that the two runs are of the same suite; it only matches ids. Check that `baseline.suite` and `candidate.suite` agree and that `missing` is empty before trusting a comparison.

## Deciding

Decide what "better" means before you look at the numbers, and write it down as a promotion rule. June has no promotion function and doesn't pick winners; it gives you the evidence, and a person decides.

A mean can hide a regression. Averaging graders, or cases, lets a gain on one hide a loss on another: a candidate that gets briefer can quietly get less accurate. So a good rule has two parts:

- **No regression** on what must never get worse: usually a deterministic grader, case by case, and on the [golden benchmark](#golden-benchmarks-and-evolving-evidence).
- **A minimum improvement** on the one metric you're trying to move.

Run the rule on the comparison, and [record it](#recording-what-was-compared) with the result. Common pieces of a rule:

- no regressed cases on the [golden benchmark](#golden-benchmarks-and-evolving-evidence);
- no grader's `delta` below zero, or below a tolerance you choose for noisy model graders;
- `failed.candidate` no higher than `failed.baseline`;
- a minimum win rate in a [pairwise preference](#wins-and-pairwise-preference).

Small suites move a lot from one case. With 20 cases, one case is 5% of a mean. Look at the cases that moved, not just the numbers.

The [Copy Agent](https://github.com/johno/june/blob/main/apps/copy-agent/README.md) works this way: `copy-agent eval` compares a candidate with the active agent on its evidence and its golden benchmark, and only `copy-agent promote`, run by a person, changes which one answers.

## Golden benchmarks and evolving evidence

Two kinds of cases answer different questions, so keep them in separate suites and never average them together:

- A **golden benchmark** is a fixed, reviewed set of cases that covers everything the app must keep doing well, including what nobody has asked for lately. It changes rarely, on purpose, in its own reviewed change, with a version you bump. A candidate shouldn't regress on it.
- **Evidence** comes from use: held-out [test splits](https://june.dev/docs/training/datasets.md#deterministic-splits) of your data, or eval cases from [chat feedback](https://june.dev/docs/chat/learning.md#eval-cases). It follows what people have been asking for, so it shifts as they do, and it's what tells you a candidate fixes real complaints.

Evidence alone lets a candidate get better at this month's requests and quietly worse at everything else. A benchmark alone never learns what people actually need. Run both, report them side by side, and gate on each.

When cases fall into groups (writing modes, topics, languages), add a group to each case's `metadata` and report the comparison per group as well, so a regression in a small group can't hide in the overall means.

The [Copy Agent](https://github.com/johno/june/blob/main/apps/copy-agent/README.md) is a working example. Its [golden benchmark](https://github.com/johno/june/blob/main/apps/copy-agent/benchmark/README.md) is a versioned file of cases by writing mode, and [`evaluate.tsx`](https://github.com/johno/june/blob/main/apps/copy-agent/src/evaluate.tsx) runs the active agent and a candidate on the evidence suite and the benchmark separately, compares each, asks a judge for a pairwise preference in both orders, and breaks the benchmark down by category.

## Reading a comparison

| Field                   |                                                                                                                  |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `baseline`, `candidate` | the two runs' target names                                                                                       |
| `scores`                | per grader: each run's mean, `delta` (the mean of per-case changes) and `paired` (how many cases `delta` covers) |
| `cases`                 | per case in both runs: `{ id, deltas }`, each grader's change, for graders that scored the case in both runs     |
| `improved`, `regressed` | ids of cases that got better or worse; see below                                                                 |
| `failed`                | how many cases had errors in each run                                                                            |
| `missing`               | ids of cases only one run has                                                                                    |

Means and deltas can disagree, and that's deliberate:

- A run's mean covers every case its grader scored in that run. A case that one run skipped or failed counts toward the other run's mean.
- `delta` only covers cases the grader scored in both runs, so a case that started failing can't make a grader look better or worse. `paired` says how many that was. When no case was scored in both runs, `delta` is missing.
- Means only cover cases that scored, so always read `failed` next to them. A candidate whose target errors on its hardest cases can show a higher mean.

A case is in `regressed` when the mean of its graders' changes is below zero, or when it had no errors in the baseline and has some in the candidate. It's in `improved` when that mean is above zero, or when its errors went away. A change in errors decides first: a case that started failing is only in `regressed`, and one whose errors went away is only in `improved`, whatever its scores did. A case with no change, or whose graders cancel out, is in neither.

Averaging a case's graders treats them all as equally important, so a gain on one grader can hide a loss on another. Check per-grader `delta`s too, or compute your own rule from `cases`, which has every change. In `compare.ts`, the candidate gets briefer and less polite on the same cases, so no case is regressed while the `polite` grader's delta is negative.

## Wins and pairwise preference

Pairwise preference is an optional second method, for qualities graders can't score well on their own. `compareEvaluations()` compares scores each target earned on its own. A judge model asked to choose between two replies to the same case is often more sensitive than two absolute scores, and gives a win rate.

That's an ordinary eval: its cases are pairs of outputs from the two runs, its target makes the choice, and its grader scores the choice:

```ts
import { openai } from "june/sdk/openai";
import { generateText, jsonSchema, Output } from "ai";
import { evalSuite, evaluate, grader, type EvalRun } from "june/train";

declare const baseline: EvalRun<string, string, string>;
declare const candidate: EvalRun<string, string, string>;

interface Pair {
  input: string;
  baseline: string;
  candidate: string;
}

const replies = new Map(candidate.cases.map((c) => [c.id, c.output]));
const pairs = baseline.cases.flatMap((left) => {
  const right = replies.get(left.id);
  return left.output !== undefined && right !== undefined
    ? [
        {
          id: left.id,
          input: { input: left.input, baseline: left.output, candidate: right },
        },
      ]
    : [];
});

async function choose(pair: Pair, [a, b]: string[], signal: AbortSignal) {
  const { output } = await generateText({
    model: openai("gpt-5-mini"),
    output: Output.object({
      schema: jsonSchema<{ winner: "A" | "B" | "tie" }>({
        type: "object",
        properties: { winner: { type: "string", enum: ["A", "B", "tie"] } },
        required: ["winner"],
        additionalProperties: false,
      }),
    }),
    prompt: `Request: ${pair.input}\n\nA: ${a}\n\nB: ${b}\n\nWhich reply is better?`,
    abortSignal: signal,
  });
  return output.winner;
}

// Ask in both orders, and call it a tie when the judge disagrees with itself.
const judge = async (pair: Pair, { signal }: { signal: AbortSignal }) => {
  const [first, second] = await Promise.all([
    choose(pair, [pair.baseline, pair.candidate], signal),
    choose(pair, [pair.candidate, pair.baseline], signal),
  ]);
  if (first === "B" && second === "A") return "candidate";
  if (first === "A" && second === "B") return "baseline";
  return "tie";
};

const preferences = await evaluate(
  judge,
  evalSuite({
    name: "support preference",
    cases: pairs,
    graders: [
      grader<Pair, string>("preference", ({ output }) => {
        if (output === "candidate") return 1;
        if (output === "baseline") return 0;
        return 0.5;
      }),
    ],
  }),
  { name: "gpt-5-mini" },
);

preferences.scores.preference?.mean; // the candidate's win rate, ties counted as half
```

Keeping the original case ids lets you line each choice up with the comparison's per-case changes. A judge is a model, so record which one it was, and evaluate both candidates again when it changes.

## Recording what was compared

`EvalRun`s and `EvalComparison`s are plain JSON. June doesn't store them; save them wherever suits you, next to anything you promote. Each run already holds every case's input, output, scores, grader `details`, `errors` and `skipped` graders, its target's name and when it ran.

A comparison is only meaningful with what it compared, so record with it:

- the suite's name and a fingerprint of its cases, such as a SHA-256 of their JSON, and the benchmark's version;
- each grader's name, what it measures, and whether it's deterministic or judged by a model;
- the judge model, for model graders and preferences;
- what each target was: a model id, a prompt version, or a fine-tune's [manifest](https://june.dev/docs/training/fine-tuning.md#the-manifest).

Then a later reader can tell whether two comparisons measured the same thing, and a candidate evaluated on an older benchmark can be evaluated again before it's promoted.

## Comparing a fine-tune

A fine-tune is one more candidate. Write the suite and run the baseline before training, on a `test` split the trainer never sees. After training, serve the artifact so it can be a target (see [Artifacts](https://june.dev/docs/training/fine-tuning.md#artifacts)), evaluate it on the same suite, and compare. If the baseline already does well, or a prompt change gets the same gain, you may not need to fine-tune at all.

In a workflow, [`<Evaluate>`](https://june.dev/docs/training/workflows.md#evaluate) under a [`<FineTune>`](https://june.dev/docs/training/workflows.md#finetune) re-runs when the fine-tune trains something new, and a function child can compare the two runs.