# Scoring and evaluation

A voice scores every draft it writes, to pick the best one. That's scoring. Whether the voice is any good, and whether a change made it better, is a separate question that takes an evaluation. This page covers both, and why the first isn't the second.

|          | Draft scoring                                 | Voice evaluation                                       |
| -------- | --------------------------------------------- | ------------------------------------------------------ |
| Question | which candidate should this request return?   | is this voice, or this version of it, actually good?   |
| Data     | the candidates and the exemplars retrieved    | held-out writing or benchmark requests, and a baseline |
| Output   | the selected draft, with every draft's scores | an evaluation run you can [compare](https://june.dev/docs/training/comparing.md)   |
| API      | evaluators, inside `voice.write()`            | `evalSuite()` and `evaluate()` from `june/train`       |

A best-scoring candidate is the best of a few drafts against the voice's own exemplars. It is never evidence that the voice is ready for production.

## Evaluators

An evaluator scores how much a text sounds like the voice:

```ts
import type { VoiceContext, VoiceScore } from "june/copy";

interface VoiceEvaluator {
  name: string;
  score(text: string, context: VoiceContext): Promise<VoiceScore>;
}
```

`context` has the `exemplars` to compare against (the samples retrieved for the request), the `request`, a `signal`, and `onCall`. A `VoiceScore` is `{ overall, dimensions, notes? }`, with `overall` and every dimension from 0 to 1. Two evaluators come with June.

## `stylometry()`

`stylometry()` compares a text's measurable habits with the exemplars' averages:

| Dimension         | Compares                                                      |
| ----------------- | ------------------------------------------------------------- |
| `sentenceLength`  | words per sentence                                            |
| `paragraphLength` | words per paragraph                                           |
| `wordLength`      | characters per word                                           |
| `contractions`    | contractions per word (`don't`, `we'll`)                      |
| `punctuation`     | how often it uses `!`, `?`, `,`, `;`, `:`, `—`, `(`, `"`, `…` |

Each dimension is 1 when the text matches the exemplars' average and falls toward 0 as one becomes a multiple of the other; `overall` is their mean. It's transparent, free and deterministic, so it's the default, a good first filter and a check on a model judge.

It also has clear limits. It knows nothing about vocabulary, content, facts or structure, so a draft can score well while saying the wrong thing at the wrong length: "sounds similar" and "does what the brief asked" are different questions, and stylometry only answers the first. And it measures against whatever was retrieved, so with the wrong exemplars it rewards sounding like the wrong writing.

## `judge()`

`judge({ model, instructions })` shows a model the exemplars and the candidate, and asks how likely it is that the same person wrote both, judging by voice and style alone (rhythm, vocabulary, punctuation, formality and humor) and ignoring the topic. It returns that likelihood as `overall` and as its one dimension, `judge`, and its one-sentence reason in `notes`. `instructions` adds guidance, such as words the person never uses:

```ts
import { openai } from "june/sdk/openai";
import { copyCorpus, judge, stylometry, voice } from "june/copy";

const fonzie = voice({
  name: "fonzie",
  corpus: copyCorpus("./fonzie.jsonl"),
  model: openai("gpt-5-mini"),
  evaluators: [
    stylometry(),
    judge({
      model: openai("gpt-5"),
      instructions: "Fonzie never uses exclamation marks or hashtags.",
    }),
  ],
});
```

A judge sees what a stylometric count can't, at the cost of a model call per draft. Its score is clamped to between 0 and 1.

## Combining evaluators

With several evaluators, `overall` is the mean of their overall scores, each dimension is prefixed with its evaluator's name (`stylometry.sentenceLength`, `judge.judge`), and every evaluator's notes are kept. With one, dimensions keep their own names.

The overall score picks the best candidate and decides whether to [revise](https://june.dev/docs/copy/voices.md#how-write-works): below the voice's `threshold` (0.75 by default), the best draft gets one rewrite that names its three weakest dimensions, kept only if it scores higher. Set `threshold: 0` to never revise.

## Scoring any text

`voice.score(text, { request })` scores any text, such as a draft a person wrote, or a competitor's copy, against the exemplars `request` retrieves:

```ts
import type { Voice } from "june/copy";

declare const fonzie: Voice;

const score = await fonzie.score("Fonzie books your interviews now.", {
  request: { type: "tweet" },
});
```

Without a request it compares against the voice's `exemplars` most recent samples (the first by id when there are no dates), so a large corpus never ends up in one prompt.

## Evaluating a voice

Scoring ranks drafts against the exemplars the voice itself picked, so it can't tell you the voice is good: a voice that retrieves the wrong samples and copies them closely scores well. Before relying on a voice, and before replacing one with another, evaluate it with [`june/train`'s evals](https://june.dev/docs/training/evals.md), and decide with [Comparing candidates](https://june.dev/docs/training/comparing.md) whether the new version wins:

- **Compare it with a baseline**: the same writer model with no voice, or the voice you have now. A score means little on its own.
- **Grade more than style.** Add graders for whether the draft does what the request asked (its form, length and facts) and whether it adds anything the person didn't give. A model judge works for these, with its reasons kept.
- **Hold out what it's graded on.** Grade on samples or requests the voice can't retrieve, or the answer is among its exemplars.
- **Read the outputs.** Scores say where to look; the drafts say why.

Here's a held-out evaluation. Cases come from the prepared `test` split, with the reconstructed brief as the request and the person's real text as `expected`; the voice is built without those samples, so it can't retrieve them:

```ts
import { openai } from "june/sdk/openai";
import { generateText } from "ai";
import {
  voice,
  type CopyRequest,
  type PreparedVoice,
  type VoiceExampleMetadata,
} from "june/copy";
import { compareEvaluations, evalSuite, evaluate, grader } from "june/train";

declare const prepared: PreparedVoice;

const held = await prepared.splits.test.examples();
const heldOut = new Set(
  held.map((example) => (example.metadata as VoiceExampleMetadata).sample),
);

const fonzie = voice({
  name: "fonzie",
  corpus: {
    ...prepared,
    samples: prepared.samples.filter((sample) => !heldOut.has(sample.id)),
  },
  model: openai("gpt-5-mini"),
});

const suite = evalSuite<CopyRequest, string, string>({
  name: "fonzie-held-out",
  cases: held.map((example) => {
    const { sample, type, inferred } = example.metadata as VoiceExampleMetadata;
    return {
      id: sample,
      input: { type: type ?? inferred.type, brief: example.prompt },
      expected: example.completion,
    };
  }),
  graders: [
    grader("voice", async ({ input, output, signal }) => {
      const score = await fonzie.score(output, { request: input, signal });
      return score.overall;
    }),
  ],
});

const withVoice = await evaluate(
  async (request: CopyRequest, { signal }) =>
    (await fonzie.write(request, { signal })).text,
  suite,
  { name: "fonzie" },
);
const generic = await evaluate(
  async (request: CopyRequest, { signal }) =>
    (
      await generateText({
        model: openai("gpt-5-mini"),
        prompt: request.brief ?? "",
        abortSignal: signal,
      })
    ).text,
  suite,
  { name: "generic" },
);

compareEvaluations(generic, withVoice).scores.voice;
```

Both runs are scored against the same voice's exemplars, so neither grades itself. Add a [judged grader](https://june.dev/docs/training/evals.md#graders) for whether each draft follows its brief, and one comparing it with `expected`, before you read anything into the result.

The common failure is a voice that wins on style and loses on the brief: it sounds more like the person than a plain writer does, while following the requested [form and length](https://june.dev/docs/copy/voices.md#form-and-length) less well. Only graders for the brief catch it. June.dev's own voice evaluation found exactly this, which is why a person still reviews every piece it drafts.

The Copy Agent's evaluation goes further, with deterministic checks per request, judged graders, a pairwise preference asked in both orders, and results broken down by kind of writing. Those graders are application code, built on `grader()` and `judge()`.

## Writing an evaluator

An evaluator is any object with a `name` and a `score` method. Return dimensions from 0 to 1, and pass every model call you make to `onCall`, as `judge()` does, so it counts toward the write's `calls` and `usage`:

```ts
import type { VoiceEvaluator } from "june/copy";

/** 1 unless the text uses a word the person never does. */
const banned = (words: string[]): VoiceEvaluator => ({
  name: "banned",
  async score(text) {
    const used = words.filter((word) =>
      new RegExp(`\\b${word}\\b`, "i").test(text),
    );
    const clean = used.length === 0 ? 1 : 0;
    return {
      overall: clean,
      dimensions: { clean },
      notes: used.length > 0 ? [`uses ${used.join(", ")}`] : undefined,
    };
  },
});
```

An evaluator's `name` is part of a voice's [manifest](https://june.dev/docs/copy/training.md#manifests), and a saved voice only loads with evaluators of the same names.