Evals

An eval measures a behavior: a set of cases, a target that answers them, and graders that score each answer. Use one to check a prompt, pick a model, test an agent, or find out whether a fine-tune actually got better. Evals don't depend on training, june/copy or a workflow; they're plain async functions you can run anywhere.

The smallest useful eval is a few cases you'd be embarrassed to get wrong and one check you can state exactly:

import { openai } from "june/sdk/openai";
import { evalSuite, evaluate, grader } from "june/train";

const suite = evalSuite<string, string, string>({
  name: "support",
  cases: [
    { id: "cancel", input: "How do I cancel my plan?", expected: "Billing" },
    { id: "invoices", input: "Where are my invoices?", expected: "Billing" },
    { id: "thanks", input: "Thanks, that fixed it!" },
  ],
  graders: [
    grader("names-the-page", ({ output, expected }) =>
      expected === undefined ? undefined : Number(output.includes(expected)),
    ),
  ],
});

const run = await evaluate(openai("gpt-5-mini"), suite);
run.scores["names-the-page"]; // { mean: 0.5, min: 0, max: 1, count: 2 }

The thanks case has nothing to point at, so its grader returns undefined and skips it rather than scoring a zero (see Graders that don't apply).

Write the suite before you change anything. Running what you have today gives a baseline, which tells you whether a change, or a fine-tune, is worth it at all and gives a candidate something to beat. An eval's output exists to support that decision: which cases a candidate fixed, which it broke, and whether to promote it. Comparing candidates covers making it.

Suites and cases

evalSuite({ name, cases, graders }) holds the cases and the graders that score each one. A case is:

interface EvalCase<I, E> {
  id?: string;
  input: I;
  expected?: E;
  metadata?: Record<string, unknown>;
}

Each case gets a stable id: its own id, or a hash of its input and expected. The same cases get the same ids on every run, so two runs can be compared case by case. Give cases an explicit id when you'll edit their wording but want to keep tracking them. Two cases with the same id, two graders with the same name, or a suite without graders are an error. name defaults to "eval".

metadata is yours: a category, a source, a reason the case exists. June keeps it on the suite's cases, where graders can read it as case.metadata.

Graders

Start with deterministic graders for what you can state exactly; use model graders only for judgments that genuinely need them.

grader(name, fn) scores one case. fn gets { input, output, expected, case, signal } and returns a number, or { score, detail } with JSON detail that's kept on the case. By convention scores run from 0 to 1, so means are comparable across graders. A score that isn't a finite number is recorded as that grader's error.

Deterministic graders

A deterministic grader is plain code: an exact match, a check that a fact or a link is present, a length limit, a schema check, word overlap with the expected answer. It's free, fast, and gives the same score every time, so prefer one whenever the property can be checked mechanically:

import { grader } from "june/train";

// Every fact listed in the case's metadata, such as a price, appears verbatim.
const keepsFacts = grader<string, string>(
  "keeps-facts",
  ({ output, case: item }) => {
    const facts = (item.metadata?.facts ?? []) as string[];
    if (facts.length === 0) return undefined;
    return facts.filter((fact) => output.includes(fact)).length / facts.length;
  },
);

const noLinks = grader<string, string>("no-links", ({ output }) =>
  /https?:\/\//.test(output) ? 0 : 1,
);

const withinLimit = grader<string, string>("within-280", ({ output }) => ({
  score: output.length <= 280 ? 1 : 0,
  detail: { length: output.length },
}));

Model graders

Some properties (helpfulness, tone, whether a reply sounds like a person) need judgment. A model grader asks another model, often called an LLM judge:

import { openai } from "june/sdk/openai";
import { generateText, Output, jsonSchema } from "ai";
import { grader } from "june/train";

const judge = grader("helpful", async ({ input, output, signal }) => {
  const { output: verdict } = await generateText({
    model: openai("gpt-5-mini"),
    output: Output.object({
      schema: jsonSchema<{ score: number; reason: string }>({
        type: "object",
        properties: { score: { type: "number" }, reason: { type: "string" } },
        required: ["score", "reason"],
        additionalProperties: false,
      }),
    }),
    prompt: `Rate from 0 to 1 how helpful this answer is.\nQuestion: ${String(input)}\nAnswer: ${String(output)}`,
    abortSignal: signal,
  });
  return { score: verdict.score, detail: verdict.reason };
});

A judge's score is a model's opinion: it costs a call per case, varies from run to run, and changes when the judge model or its prompt does. Pass the signal so aborting a run stops its calls, keep the judge's reason in detail so a score can be checked, use the same judge for every run you'll compare, and record which model it was. Mix both kinds in one suite: deterministic graders for what can be checked, a judge for what can't.

Graders that don't apply

A grader that doesn't apply to a case returns undefined. The case gets no score from it and no error, and the grader is listed in the case's skipped. One suite can then mix cases, such as replies with a known fix and replies without:

import { grader } from "june/train";

const matches = grader<string, string, string>(
  "matches",
  ({ output, expected }) =>
    expected === undefined
      ? undefined
      : Number(output.trim() === expected.trim()),
);

So a missing score always says why: a low score is a number, a failure is in errors, and a grader that doesn't apply is in skipped. A skip isn't a zero: it leaves the case out of that grader's mean and count, and out of its comparisons.

Targets

evaluate(target, suite, options) runs every case's input through target, then every grader on the output. A target is:

• an AI SDK language model. The input is a prompt string, messages, or { system, prompt } / { system, messages }, and the output is the generated text. Any model works: a provider's, a tuned provider model, or a locally served adapter.
• a function (input, { signal }) => output, sync or async, for anything else: an agent, a pipeline, a June workflow, or a model behind a server you call yourself.

A target or grader that throws doesn't fail the run. Its error is recorded on the case, with source set to "target" or the grader's name, and that score is left out. When the target fails, no grader runs for that case. A grader that returns undefined isn't an error: the case doesn't count toward failed.

Results

evaluate() resolves with an EvalRun:

import type { EvalCaseResult, ScoreSummary } from "june/train";

interface EvalRun {
  suite: string;
  target: string;
  cases: EvalCaseResult[]; // in suite order: id, input, expected, output, scores, details, errors, skipped, durationMs
  scores: Record<string, ScoreSummary>; // per grader: mean, min, max, count
  failed: number; // cases with any error
  startedAt: string;
  durationMs: number;
}

A grader's count is how many cases it scored, so a mean over fewer cases than the suite has is easy to spot. Cases it skipped or failed on don't count, and a grader that scored no case has no entry in scores.

A run is plain JSON, apart from whatever your target returns, so it can be saved and read later. To find out why a score moved, read the cases, not the means: each case has its output, every grader's score and details (a judge's reason, a length), its errors with their source, and the graders it skipped. June doesn't store runs; keep the ones you'll compare against, and record what produced them (see Recording what was compared).

Comparing runs

compareEvaluations(baseline, candidate) lines two runs of the same suite up by case id, and reports each grader's change and which cases improved or regressed:

import { compareEvaluations, type EvalRun } from "june/train";

declare const baseline: EvalRun;
declare const candidate: EvalRun;

const comparison = compareEvaluations(baseline, candidate);
comparison.scores["names-the-page"]; // { baseline: 0.5, candidate: 1, delta: 0.5, paired: 2 }
comparison.regressed; // ids of cases that got worse

How to read it, and how to decide whether a candidate is better, is the subject of Comparing candidates.

Dataset-backed cases

Hand-written cases are the place to start. Once you have data, a held-out split of it makes the cases:

import { casesFrom, dataset, evalSuite, grader } from "june/train";

const { test } = dataset("./support.jsonl").split({
  train: 0.9,
  test: 0.1,
  seed: 42,
});

const suite = evalSuite({
  name: "support",
  cases: await casesFrom(test),
  graders: [
    grader("brevity", ({ output }) => (String(output).length <= 140 ? 1 : 0)),
  ],
});

casesFrom(dataset) turns a dataset into cases, keeping each example's metadata. A completion's prompt is the input and its completion is expected. A conversation's messages up to its last assistant turn are the input, and that turn's text is expected; a conversation whose last assistant turn isn't plain text, or has nothing before it, is skipped. Preference examples are skipped. Hold out a test split for this, so you don't evaluate on what you trained on. For cases from chat feedback, see Learning from chat.

Record the test split's fingerprint next to the runs, so you can show what they were measured on.

In a workflow

evaluate() is a plain async function, so it runs in a June workflow with useStep, and its result lands in the run's steps. <Evaluate> is the same as a JSX step, with progress on the trace and a cache key that follows a <FineTune> above it.

import { openai } from "june/sdk/openai";
import { useStep } from "june";
import { evaluate, type EvalSuite } from "june/train";

declare const suite: EvalSuite;

export default function Evals() {
  useStep("support-evals", () => evaluate(openai("gpt-5-mini"), suite));
  return null;
}