# Evals and training

**Evaluate first. Train only when the evidence says you should.**

`june/train` gives you the pieces to build datasets, run repeatable evals, compare a candidate with what's active, and, when prompting, retrieval or examples are no longer enough, orchestrate a fine-tune. Most improvements never need the last step: a better prompt or a better retrieval set is a candidate too, and it's judged on the same evals.

```text
dataset → eval → compare → (fine-tune → eval again) → a person promotes
```

1. **Dataset.** Collect examples of the behavior, from files or from [chat feedback](https://june.dev/docs/chat/learning.md), and split off a test set before anything else.
2. **Eval.** Turn the behavior into cases and graders, and run what's active today: the baseline.
3. **Compare.** Run a candidate on the same cases and line it up with the baseline case by case.
4. **Fine-tune, only if needed.** When cheaper candidates plateau, train one, then evaluate it on the same held-out cases.
5. **Promote.** A person reads the comparison and decides. June gives you the evidence and lineage as plain values to keep; it doesn't store eval runs or promote anything itself.

## Use as much as you need

`june/train` doesn't depend on `june/copy`, `june/chat` or a workflow, and each layer works without the next:

- **Evals only.** `evalSuite()`, `evaluate()` and `compareEvaluations()` score any model, agent or function. No dataset or trainer needed.
- **Datasets and evals.** `dataset()` loads, cleans and splits examples, and `casesFrom()` turns a held-out split into eval cases.
- **Fine-tuning.** `fineTune()` adds a trainer: `trl()`, `vertex()` or your own.
- **In workflows.** `<FineTune>` and `<Evaluate>` run the same calls as traced, cancelable workflow steps.

[`june/copy`](https://june.dev/docs/copy.md) and [learning from chat](https://june.dev/docs/chat/learning.md) are built on these primitives, and the [Copy Agent](https://github.com/johno/june/blob/main/apps/copy-agent/README.md) is a complete application that uses them: it turns chat feedback into evidence and evaluates candidates against a golden benchmark, and a person promotes a candidate only after seeing that evaluation.

## Architecture

June **orchestrates training; it isn't an ML framework.** It owns the data, the job lifecycle, the evals and the record of what was trained. The weights, GPUs, optimization and provider APIs belong to a trainer. June doesn't decide when to train, tune anything on its own, or promise that a fine-tune behaves the same on every trainer.

```text
              june/train                                 june/trainers
┌───────────────────────────────────────┐        ┌──────────────────────────┐
│ dataset()  ──split──▶ train / test    │        │ trl()    → Python + TRL  │
│                                       │ Trainer│ vertex() → Vertex AI     │
│ fineTune() ── start/status/cancel ────┼───────▶│ yours    → anything      │
│    │                                  │        └──────────────────────────┘
│    ▼                                  │
│ Job ── ref (JSON) ── survives restarts│
│    │                                  │
│    ▼                                  │
│ artifact + manifest                   │
│                                       │
│ evalSuite() ─▶ evaluate() ─▶ compare  │
│                                       │
│ <FineTune> / <Evaluate> in workflows  │
└───────────────────────────────────────┘
                   ▲
                   │ built on
┌───────────────────────────────────────┐
│ june/copy: corpus → prepareVoice()    │
│            → voice().write()          │
└───────────────────────────────────────┘
```

| June owns                                      | Trainers and providers own                     |
| ---------------------------------------------- | ---------------------------------------------- |
| canonical datasets, fingerprints and splits    | converting examples to their format            |
| checking a trainer supports a method           | running the training, on whatever hardware     |
| starting, polling and canceling jobs           | the job's real state and its progress          |
| a JSON manifest of what was trained, from what | the weights, endpoints and provider provenance |
| eval suites, runs and comparisons              | serving a tuned model                          |
| copy corpora, reverse instructions, voices     |                                                |

## Local or managed

The same `fineTune()` call targets either. Only the trainer changes:

|          | `trl()` (local)                                                                  | `vertex()` (managed)                                 |
| -------- | -------------------------------------------------------------------------------- | ---------------------------------------------------- |
| Runs     | a Python process on your machine                                                 | Google's infrastructure                              |
| Models   | open-weight models from Hugging Face                                             | Gemini models Vertex can tune                        |
| Methods  | `sft`, `lora`, `qlora`                                                           | `sft`                                                |
| Artifact | `adapter` or `checkpoint` files you own                                          | a `provider-model` endpoint inside Vertex            |
| Data     | stays on your machine                                                            | uploaded to your Cloud Storage bucket                |
| Config   | `epochs`, `learningRate`, `batchSize`, `maxSequenceLength`, `seed`, LoRA options | `epochs`; the rest through `providerOptions.vertex`  |
| Examples | `completion` or `conversation`, one shape per dataset                            | `completion` and `conversation`, text only           |
| Needs    | uv, and a GPU for anything but small models                                      | a Google Cloud project, a bucket and an access token |

The call is portable; the results aren't. The same dataset and method on two trainers produce different artifacts, trained differently, and need to be served and evaluated separately. Compare them on the same eval suite rather than assuming one stands in for the other.

## Artifacts

A fine-tune produces an artifact, which isn't always a model you can call:

- `adapter`: weights on top of the base model, such as a LoRA. Serve it with a LoRA-capable server.
- `checkpoint`: a full set of weights.
- `provider-model`: a model that only exists inside a provider, called by its `modelId`.

Every fine-tune also returns a [manifest](https://june.dev/docs/training/fine-tuning.md#the-manifest): plain JSON naming the trainer, base model, method, config and the fingerprints of the data it trained on. Store it next to the artifact and next to the comparison that justified promoting it.

## Guides

Read them in order the first time. The first three are all most agents need:

1. [Datasets](https://june.dev/docs/training/datasets.md): splits that keep test data out of training, fingerprints, example shapes, transforms
2. [Evals](https://june.dev/docs/training/evals.md): the smallest useful eval, deterministic and model graders, results
3. [Comparing candidates](https://june.dev/docs/training/comparing.md): promotion rules, golden benchmarks and evolving evidence
4. [Fine-tuning](https://june.dev/docs/training/fine-tuning.md): when to fine-tune and when not to, `fineTune()`, artifacts, manifests, serving
5. [Jobs](https://june.dev/docs/training/jobs.md): work a backend owns, which outlives your process
6. [Training in workflows](https://june.dev/docs/training/workflows.md): when `<FineTune>` and `<Evaluate>` are worth it over plain calls
7. [Local training](https://june.dev/docs/training/local.md): hardware, TRL and PEFT with uv
8. [Managed training](https://june.dev/docs/training/managed.md): Vertex AI, its data and credential boundary, swapping trainers
9. [Custom trainers](https://june.dev/docs/training/custom-trainers.md): when to write one, and the smallest contract

Related:

- [Learning from chat](https://june.dev/docs/chat/learning.md): feedback into datasets and eval cases
- [Copy and voice](https://june.dev/docs/copy.md): from a person's writing to a voice, its drafts and their edits, with guides to [corpora](https://june.dev/docs/copy/corpora.md), [preparation](https://june.dev/docs/copy/preparation.md), [writing](https://june.dev/docs/copy/voices.md), [retrieval](https://june.dev/docs/copy/retrieval.md), [scoring](https://june.dev/docs/copy/scoring.md), [saving and training](https://june.dev/docs/copy/training.md) and [learning from edits](https://june.dev/docs/copy/learning.md)
- [Chat](https://june.dev/docs/chat.md): sessions, turns, responders, storage, context and feedback
- [Tools](https://june.dev/docs/workflows/tools.md): `<Bash />` and where it runs
- [Security and privacy](https://june.dev/docs/reference/security.md)
- [Compatibility](https://june.dev/docs/reference/compatibility.md): what's stable

## Examples

Each one runs; the first four need nothing but Bun, and June's tests run them.

| Example                                                           |                                                                  |
| ----------------------------------------------------------------- | ---------------------------------------------------------------- |
| [`quickstart.ts`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/quickstart.ts)         | dataset, baseline eval, a custom trainer, tuned eval, comparison |
| [`compare.ts`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/compare.ts)               | two candidates on a golden benchmark and held-out evidence       |
| [`voice.ts`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/voice.ts)                   | prepare a corpus, train a voice, write with it                   |
| [`workflow.tsx`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/workflow.tsx)           | `<FineTune>` then `<Evaluate>` under `june run`                  |
| [`dataset.ts`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/dataset.ts)               | load, filter, dedupe and split a JSONL dataset                   |
| [`local-trl.ts`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/local-trl.ts)           | LoRA-tune a small model with TRL (needs uv)                      |
| [`managed-vertex.ts`](https://github.com/johno/june/blob/main/packages/june/src/examples/training/managed-vertex.ts) | tune Gemini on Vertex AI (needs a Google Cloud project)          |

## Imports

```ts
import { dataset, evaluate, fineTune, FineTune } from "june/train";
import { trl, vertex } from "june/trainers";
import { copyCorpus, prepareVoice, voice } from "june/copy";
```

All three are subpaths of the `june` package, with no npm dependencies beyond June's own. Python's ML packages are only ever installed in the TRL runner's isolated environment.