# Evals

An eval measures a behavior: a set of cases, a target that answers them, and graders that score each answer. Use one to check a prompt, pick a model, test an agent, or find out whether a fine-tune actually got better. Evals don't depend on training, `june/copy` or a workflow; they're plain async functions you can run anywhere.

The smallest useful eval is a few cases you'd be embarrassed to get wrong and one check you can state exactly:

```ts
import { openai } from "june/sdk/openai";
import { evalSuite, evaluate, grader } from "june/train";

const suite = evalSuite<string, string, string>({
  name: "support",
  cases: [
    { id: "cancel", input: "How do I cancel my plan?", expected: "Billing" },
    { id: "invoices", input: "Where are my invoices?", expected: "Billing" },
    { id: "thanks", input: "Thanks, that fixed it!" },
  ],
  graders: [
    grader("names-the-page", ({ output, expected }) =>
      expected === undefined ? undefined : Number(output.includes(expected)),
    ),
  ],
});

const run = await evaluate(openai("gpt-5-mini"), suite);
run.scores["names-the-page"]; // { mean: 0.5, min: 0, max: 1, count: 2 }
```

The `thanks` case has nothing to point at, so its grader returns `undefined` and skips it rather than scoring a zero (see [Graders that don't apply](#graders-that-dont-apply)).

Write the suite before you change anything. Running what you have today gives a baseline, which tells you whether a change, or a fine-tune, is worth it at all and gives a candidate something to beat. An eval's output exists to support that decision: which cases a candidate fixed, which it broke, and whether to promote it. [Comparing candidates](https://june.dev/docs/training/comparing.md) covers making it.

## Suites and cases

`evalSuite({ name, cases, graders })` holds the cases and the graders that score each one. A case is:

```ts
interface EvalCase<I, E> {
  id?: string;
  input: I;
  expected?: E;
  metadata?: Record<string, unknown>;
}
```

Each case gets a stable id: its own `id`, or a hash of its `input` and `expected`. The same cases get the same ids on every run, so two runs can be compared case by case. Give cases an explicit `id` when you'll edit their wording but want to keep tracking them. Two cases with the same id, two graders with the same name, or a suite without graders are an error. `name` defaults to `"eval"`.

`metadata` is yours: a category, a source, a reason the case exists. June keeps it on the suite's cases, where graders can read it as `case.metadata`.

## Graders

Start with deterministic graders for what you can state exactly; use model graders only for judgments that genuinely need them.

`grader(name, fn)` scores one case. `fn` gets `{ input, output, expected, case, signal }` and returns a number, or `{ score, detail }` with JSON detail that's kept on the case. By convention scores run from 0 to 1, so means are comparable across graders. A score that isn't a finite number is recorded as that grader's error.

### Deterministic graders

A deterministic grader is plain code: an exact match, a check that a fact or a link is present, a length limit, a schema check, word overlap with the expected answer. It's free, fast, and gives the same score every time, so prefer one whenever the property can be checked mechanically:

```ts
import { grader } from "june/train";

// Every fact listed in the case's metadata, such as a price, appears verbatim.
const keepsFacts = grader<string, string>(
  "keeps-facts",
  ({ output, case: item }) => {
    const facts = (item.metadata?.facts ?? []) as string[];
    if (facts.length === 0) return undefined;
    return facts.filter((fact) => output.includes(fact)).length / facts.length;
  },
);

const noLinks = grader<string, string>("no-links", ({ output }) =>
  /https?:\/\//.test(output) ? 0 : 1,
);

const withinLimit = grader<string, string>("within-280", ({ output }) => ({
  score: output.length <= 280 ? 1 : 0,
  detail: { length: output.length },
}));
```

### Model graders

Some properties (helpfulness, tone, whether a reply sounds like a person) need judgment. A model grader asks another model, often called an LLM judge:

```ts
import { openai } from "june/sdk/openai";
import { generateText, Output, jsonSchema } from "ai";
import { grader } from "june/train";

const judge = grader("helpful", async ({ input, output, signal }) => {
  const { output: verdict } = await generateText({
    model: openai("gpt-5-mini"),
    output: Output.object({
      schema: jsonSchema<{ score: number; reason: string }>({
        type: "object",
        properties: { score: { type: "number" }, reason: { type: "string" } },
        required: ["score", "reason"],
        additionalProperties: false,
      }),
    }),
    prompt: `Rate from 0 to 1 how helpful this answer is.\nQuestion: ${String(input)}\nAnswer: ${String(output)}`,
    abortSignal: signal,
  });
  return { score: verdict.score, detail: verdict.reason };
});
```

A judge's score is a model's opinion: it costs a call per case, varies from run to run, and changes when the judge model or its prompt does. Pass the `signal` so aborting a run stops its calls, keep the judge's reason in `detail` so a score can be checked, use the same judge for every run you'll compare, and record which model it was. Mix both kinds in one suite: deterministic graders for what can be checked, a judge for what can't.

### Graders that don't apply

A grader that doesn't apply to a case returns `undefined`. The case gets no score from it and no error, and the grader is listed in the case's `skipped`. One suite can then mix cases, such as replies with a known fix and replies without:

```ts
import { grader } from "june/train";

const matches = grader<string, string, string>(
  "matches",
  ({ output, expected }) =>
    expected === undefined
      ? undefined
      : Number(output.trim() === expected.trim()),
);
```

So a missing score always says why: a low score is a number, a failure is in `errors`, and a grader that doesn't apply is in `skipped`. A skip isn't a zero: it leaves the case out of that grader's mean and `count`, and out of its comparisons.

## Targets

`evaluate(target, suite, options)` runs every case's input through `target`, then every grader on the output. A target is:

- an AI SDK language model. The input is a prompt string, messages, or `{ system, prompt }` / `{ system, messages }`, and the output is the generated text. Any model works: a provider's, a tuned provider model, or a locally served adapter.
- a function `(input, { signal }) => output`, sync or async, for anything else: an agent, a pipeline, a June workflow, or a model behind a server you call yourself.

| Option        |                                                                                                  |
| ------------- | ------------------------------------------------------------------------------------------------ |
| `name`        | the target's name in results; defaults to `provider:modelId` for a model, or the function's name |
| `concurrency` | how many cases run at once (4 by default)                                                        |
| `signal`      | rejects the run with the abort reason                                                            |
| `onCase`      | called as each case finishes, with its result and how many of the total are done                 |

A target or grader that throws doesn't fail the run. Its error is recorded on the case, with `source` set to `"target"` or the grader's name, and that score is left out. When the target fails, no grader runs for that case. A grader that returns `undefined` isn't an error: the case doesn't count toward `failed`.

## Results

`evaluate()` resolves with an `EvalRun`:

```ts
import type { EvalCaseResult, ScoreSummary } from "june/train";

interface EvalRun {
  suite: string;
  target: string;
  cases: EvalCaseResult[]; // in suite order: id, input, expected, output, scores, details, errors, skipped, durationMs
  scores: Record<string, ScoreSummary>; // per grader: mean, min, max, count
  failed: number; // cases with any error
  startedAt: string;
  durationMs: number;
}
```

A grader's `count` is how many cases it scored, so a mean over fewer cases than the suite has is easy to spot. Cases it skipped or failed on don't count, and a grader that scored no case has no entry in `scores`.

A run is plain JSON, apart from whatever your target returns, so it can be saved and read later. To find out why a score moved, read the cases, not the means: each case has its `output`, every grader's score and `details` (a judge's reason, a length), its `errors` with their `source`, and the graders it `skipped`. June doesn't store runs; keep the ones you'll compare against, and record what produced them (see [Recording what was compared](https://june.dev/docs/training/comparing.md#recording-what-was-compared)).

## Comparing runs

`compareEvaluations(baseline, candidate)` lines two runs of the same suite up by case id, and reports each grader's change and which cases improved or regressed:

```ts
import { compareEvaluations, type EvalRun } from "june/train";

declare const baseline: EvalRun;
declare const candidate: EvalRun;

const comparison = compareEvaluations(baseline, candidate);
comparison.scores["names-the-page"]; // { baseline: 0.5, candidate: 1, delta: 0.5, paired: 2 }
comparison.regressed; // ids of cases that got worse
```

How to read it, and how to decide whether a candidate is better, is the subject of [Comparing candidates](https://june.dev/docs/training/comparing.md).

## Dataset-backed cases

Hand-written cases are the place to start. Once you have data, a held-out split of it makes the cases:

```ts
import { casesFrom, dataset, evalSuite, grader } from "june/train";

const { test } = dataset("./support.jsonl").split({
  train: 0.9,
  test: 0.1,
  seed: 42,
});

const suite = evalSuite({
  name: "support",
  cases: await casesFrom(test),
  graders: [
    grader("brevity", ({ output }) => (String(output).length <= 140 ? 1 : 0)),
  ],
});
```

`casesFrom(dataset)` turns a dataset into cases, keeping each example's metadata. A completion's prompt is the input and its completion is expected. A conversation's messages up to its last assistant turn are the input, and that turn's text is expected; a conversation whose last assistant turn isn't plain text, or has nothing before it, is skipped. Preference examples are skipped. Hold out a `test` split for this, so you don't evaluate on what you trained on. For cases from chat feedback, see [Learning from chat](https://june.dev/docs/chat/learning.md#eval-cases).

Record the test split's [fingerprint](https://june.dev/docs/training/datasets.md#fingerprints) next to the runs, so you can show what they were measured on.

## In a workflow

`evaluate()` is a plain async function, so it runs in a June workflow with `useStep`, and its result lands in the run's steps. [`<Evaluate>`](https://june.dev/docs/training/workflows.md#evaluate) is the same as a JSX step, with progress on the trace and a cache key that follows a `<FineTune>` above it.

```tsx
import { openai } from "june/sdk/openai";
import { useStep } from "june";
import { evaluate, type EvalSuite } from "june/train";

declare const suite: EvalSuite;

export default function Evals() {
  useStep("support-evals", () => evaluate(openai("gpt-5-mini"), suite));
  return null;
}
```