# Learning from chat

Real conversations are the best evidence of what a model or agent gets wrong. `june/train` turns chat sessions, and the feedback people left on them, into datasets and eval cases. Conversion only reads: it makes no model calls and never writes to a session's journal.

Used well, that evidence goes through one loop:

```text
sessions ─▶ curated feedback ─▶ datasets and eval cases ─▶ candidate ─▶ held-out evaluation ─▶ a person promotes it
                (approved                                   (a fine-tune or
                 replies and edits)                          a changed agent)
```

> **A reply isn't training data just because it happened.** Training on every reply teaches a model to repeat its own mistakes, so by default only replies a person approved (a positive rating) or corrected (a preferred edit) become training targets. An `include` function can change that rule, and the choice is yours.

Feedback is recorded with [`session.feedback()`](https://june.dev/docs/chat/feedback.md). `june/train` reads `june/chat` sessions; `june/chat` never depends on `june/train`. The [Copy Agent](https://github.com/johno/june/blob/main/apps/copy-agent/README.md) is a complete application of this loop: edits in chat, evidence from `june/train`, a candidate evaluated against the active agent, and promotion by a person.

## Curation

The converters' defaults decide which turns are evidence, and for what:

| Turn                                | Conversation dataset             | Preference dataset             | Eval cases                               |
| ----------------------------------- | -------------------------------- | ------------------------------ | ---------------------------------------- |
| has a preferred edit                | the edit is the target           | edit chosen, original rejected | `expected` is the edit                   |
| latest rating `"positive"`, no edit | the original reply is the target | —                              | with `positive: true`, a regression case |
| latest rating `"negative"`, no edit | left out                         | —                              | with `negative: true`, no `expected`     |
| no feedback, or a numeric rating    | left out                         | —                              | left out                                 |
| failed, canceled or interrupted     | left out                         | —                              | left out                                 |

When a turn has several ratings or edits, the latest of each counts; the earlier ones stay in the journal. Ratings judge the original reply, so a negative rating, even one recorded after an edit, never discards the edit. A negative reply without a correction says what's wrong but not what's right, so it's eval material, never a training target.

## Conversation datasets

```ts
import { createChat, fileStore, type Responder } from "june/chat";
import { datasetFromSessions } from "june/train";

declare const responder: Responder;

const chat = createChat({ responder, store: fileStore(".june/chat") });
const sessions = await Promise.all(
  ["4f9c0e1a", "b2d1aa07"].map((id) => chat.openSession(id)),
);

const sft = datasetFromSessions(sessions);
console.log(await sft.summary());
```

Each eligible turn becomes one `conversation` example: the completed turns before it, the user's input, and the target reply. Earlier turns appear as the full transcript, each with its steering and its original reply, as `fullContext()` would send them, so the example teaches the right answer to the conversation as it actually happened. If the chat used `summaryContext()` or a custom context, that's more than the responder saw.

Pass `include` to curate differently. It receives each completed turn with its `context`, latest `rating` and latest `preferred` reply, and the target is still the edit when there is one:

```ts
import type { Session } from "june/chat";
import { datasetFromSessions } from "june/train";

declare const sessions: Session[];

const toneFixes = datasetFromSessions(sessions, {
  include: ({ turn, preferred }) =>
    preferred !== undefined &&
    turn.feedback.some((feedback) => feedback.tags?.includes("tone")),
});
```

## Preference datasets

```ts
import type { Session } from "june/chat";
import { preferenceDatasetFromSessions } from "june/train";

declare const sessions: Session[];

const prefs = preferenceDatasetFromSessions(sessions);
```

Every turn someone edited becomes a `preference` example: `input` is the conversation before the reply, `chosen` the latest edit, and `rejected` the original reply. An edit that doesn't change the reply is skipped.

## Eval cases

```ts
import type { Session } from "june/chat";
import { evalCasesFromSessions, evalSuite, grader } from "june/train";

declare const sessions: Session[];

const suite = evalSuite({
  name: "chat-regressions",
  cases: evalCasesFromSessions(sessions, { negative: true }),
  graders: [
    grader<unknown, string, string>("short", ({ output }) =>
      output.length < 400 ? 1 : 0,
    ),
  ],
});
```

Each case's `id` is `chat:<session id>:<turn id>`, stable across reruns and edits, so runs line up in `compareEvaluations()`. A case's `input` is the conversation up to the reply, as AI SDK messages, so it runs against a model target directly. Edited turns expect the edit. Negatively rated turns without an edit only become cases with `negative: true`, and have no `expected`: June won't invent the right answer, so grade them on their own terms. `positive: true` adds approved replies as regression cases that expect the original.

## Evaluate, then train only if it's justified

The outputs are ordinary datasets and cases, so they go wherever any other dataset does. Hold some sessions out first: a model graded on the turns it trained on shows whether it learned them, not whether it generalizes.

```ts
import { openai } from "june/sdk/openai";
import type { Session } from "june/chat";
import {
  datasetFromSessions,
  evalCasesFromSessions,
  evalSuite,
  evaluate,
  fineTune,
  grader,
  type Trainer,
} from "june/train";

declare const sessions: Session[];
declare const trainer: Trainer;

// Evaluate only on sessions training never sees.
const heldOut = sessions.slice(-5);
const training = sessions.slice(0, -5);

const suite = evalSuite({
  name: "chat",
  cases: evalCasesFromSessions(heldOut),
  graders: [
    grader<unknown, string, string>("matches-edit", ({ output, expected }) =>
      output.trim() === expected?.trim() ? 1 : 0,
    ),
  ],
});

// The model that answers these chats today.
const active = await evaluate(openai("gpt-5-mini"), suite);
console.log(active.scores);

// Only if it falls short in a way more examples can fix: tune the
// open-weight model you'd serve instead, then evaluate it on the same suite.
const { artifact, manifest } = await fineTune({
  model: "Qwen/Qwen3-8B",
  trainer,
  dataset: datasetFromSessions(training).split({
    train: 0.9,
    test: 0.1,
    seed: 7,
  }),
  method: "lora",
});
console.log(artifact.type, manifest.datasets.train.fingerprint);
```

- **Evaluate first.** The active model's scores on held-out cases say whether there's a problem worth training for. A better prompt, context or agent may fix it without a fine-tune; see [Fine-tuning](https://june.dev/docs/training/fine-tuning.md).
- **Keep evaluation and training apart.** Build eval cases only from sessions, or turns, that training never sees. [Datasets](https://june.dev/docs/training/datasets.md) covers deterministic splits.
- **Compare on the same cases.** Evaluate the candidate, a tuned model or a changed agent, on the same suite and compare the two runs with `compareEvaluations()`. [Comparing candidates](https://june.dev/docs/training/comparing.md) covers deciding what "better" means before you look.

The same stored sessions always produce the same examples, so a dataset's fingerprint identifies exactly which conversations a model was trained on.

## Review before promotion

Nothing here changes what answers your chats. A fine-tune produces an artifact and an eval produces scores; deciding a candidate is better, and switching a chat's model or responder to it, is a deliberate step a person takes. June doesn't promote models, rewrite agents, or retrain on its own.

Before training, read what you're about to train on: feedback is written by people, edits can carry mistakes or private data, and a session's content went to whatever model answered it.

## Provenance

Every example and case carries `metadata` saying where it came from:

```json
{
  "source": "chat",
  "sessionId": "4f9c0e1a",
  "turnId": "7c3e91d2",
  "at": "2026-10-14T16:12:03.512Z",
  "responder": { "kind": "model" },
  "model": { "provider": "openai.responses", "modelId": "gpt-5-mini" },
  "tags": ["verbosity"],
  "target": "preferred"
}
```

`target` says whether the example teaches the human's edit or the original reply. Keep metadata when you write datasets out, so you can trace a tuned model's behavior back to the conversations it learned from.