Evals and training

Evaluate first. Train only when the evidence says you should.

june/train gives you the pieces to build datasets, run repeatable evals, compare a candidate with what's active, and, when prompting, retrieval or examples are no longer enough, orchestrate a fine-tune. Most improvements never need the last step: a better prompt or a better retrieval set is a candidate too, and it's judged on the same evals.

dataset → eval → compare → (fine-tune → eval again) → a person promotes

1. Dataset. Collect examples of the behavior, from files or from chat feedback, and split off a test set before anything else.
2. Eval. Turn the behavior into cases and graders, and run what's active today: the baseline.
3. Compare. Run a candidate on the same cases and line it up with the baseline case by case.
4. Fine-tune, only if needed. When cheaper candidates plateau, train one, then evaluate it on the same held-out cases.
5. Promote. A person reads the comparison and decides. June gives you the evidence and lineage as plain values to keep; it doesn't store eval runs or promote anything itself.

Use as much as you need

june/train doesn't depend on june/copy, june/chat or a workflow, and each layer works without the next:

• Evals only. evalSuite(), evaluate() and compareEvaluations() score any model, agent or function. No dataset or trainer needed.
• Datasets and evals. dataset() loads, cleans and splits examples, and casesFrom() turns a held-out split into eval cases.
• Fine-tuning. fineTune() adds a trainer: trl(), vertex() or your own.
• In workflows. <FineTune> and <Evaluate> run the same calls as traced, cancelable workflow steps.

june/copy and learning from chat are built on these primitives, and the Copy Agent is a complete application that uses them: it turns chat feedback into evidence and evaluates candidates against a golden benchmark, and a person promotes a candidate only after seeing that evaluation.

Architecture

June orchestrates training; it isn't an ML framework. It owns the data, the job lifecycle, the evals and the record of what was trained. The weights, GPUs, optimization and provider APIs belong to a trainer. June doesn't decide when to train, tune anything on its own, or promise that a fine-tune behaves the same on every trainer.

              june/train                                 june/trainers
┌───────────────────────────────────────┐        ┌──────────────────────────┐
│ dataset()  ──split──▶ train / test    │        │ trl()    → Python + TRL  │
│                                       │ Trainer│ vertex() → Vertex AI     │
│ fineTune() ── start/status/cancel ────┼───────▶│ yours    → anything      │
│    │                                  │        └──────────────────────────┘
│    ▼                                  │
│ Job ── ref (JSON) ── survives restarts│
│    │                                  │
│    ▼                                  │
│ artifact + manifest                   │
│                                       │
│ evalSuite() ─▶ evaluate() ─▶ compare  │
│                                       │
│ <FineTune> / <Evaluate> in workflows  │
└───────────────────────────────────────┘
                   ▲
                   │ built on
┌───────────────────────────────────────┐
│ june/copy: corpus → prepareVoice()    │
│            → voice().write()          │
└───────────────────────────────────────┘

Local or managed

The same fineTune() call targets either. Only the trainer changes:

The call is portable; the results aren't. The same dataset and method on two trainers produce different artifacts, trained differently, and need to be served and evaluated separately. Compare them on the same eval suite rather than assuming one stands in for the other.

Artifacts

A fine-tune produces an artifact, which isn't always a model you can call:

• adapter: weights on top of the base model, such as a LoRA. Serve it with a LoRA-capable server.
• checkpoint: a full set of weights.
• provider-model: a model that only exists inside a provider, called by its modelId.

Every fine-tune also returns a manifest: plain JSON naming the trainer, base model, method, config and the fingerprints of the data it trained on. Store it next to the artifact and next to the comparison that justified promoting it.

Guides

Read them in order the first time. The first three are all most agents need:

1. Datasets: splits that keep test data out of training, fingerprints, example shapes, transforms
2. Evals: the smallest useful eval, deterministic and model graders, results
3. Comparing candidates: promotion rules, golden benchmarks and evolving evidence
4. Fine-tuning: when to fine-tune and when not to, fineTune(), artifacts, manifests, serving
5. Jobs: work a backend owns, which outlives your process
6. Training in workflows: when <FineTune> and <Evaluate> are worth it over plain calls
7. Local training: hardware, TRL and PEFT with uv
8. Managed training: Vertex AI, its data and credential boundary, swapping trainers
9. Custom trainers: when to write one, and the smallest contract

Related:

• Learning from chat: feedback into datasets and eval cases
• Copy and voice: from a person's writing to a voice, its drafts and their edits, with guides to corpora, preparation, writing, retrieval, scoring, saving and training and learning from edits
• Chat: sessions, turns, responders, storage, context and feedback
• Tools: <Bash /> and where it runs
• Security and privacy
• Compatibility: what's stable

Examples

Each one runs; the first four need nothing but Bun, and June's tests run them.

Imports

import { dataset, evaluate, fineTune, FineTune } from "june/train";
import { trl, vertex } from "june/trainers";
import { copyCorpus, prepareVoice, voice } from "june/copy";

All three are subpaths of the june package, with no npm dependencies beyond June's own. Python's ML packages are only ever installed in the TRL runner's isolated environment.