# Datasets A `Dataset` is an immutable, ordered list of canonical examples: the data you train on and the data you hold out to [evaluate](https://june.dev/docs/training/evals.md) with. It's the same for every trainer: converting examples to a provider's format is the trainer's job, not the dataset's. Datasets don't need a trainer, a model or `june/copy`; you can use them for evals alone. A dataset can feed training or evaluation, but the same examples shouldn't do both. Split first, keep a test set untouched, and record fingerprints so you can prove which data a candidate saw: ```ts import { dataset } from "june/train"; const { train, validation, test } = dataset("./support.jsonl") .filter((example) => example.type === "completion") .dedupe() .split({ train: 0.8, validation: 0.1, test: 0.1, seed: 42 }); await test.fingerprint(); // "sha256:…", recorded next to every eval run on it ``` - **`train`** is what a trainer learns from, or the examples a prompt or voice draws on. - **`validation`** is what a trainer scores itself on during training, such as `trl()` each epoch, to show over-fitting. It isn't a benchmark. - **`test`** is held out from everything. It becomes your [eval cases](https://june.dev/docs/training/evals.md#suites-and-cases), and its scores are the ones you [compare candidates](https://june.dev/docs/training/comparing.md) on. Splits are deterministic, so the same test set comes back on every machine and every run, and a fingerprint proves it hasn't changed. Both are what make an evaluation trustworthy, not bookkeeping. ## Example shapes ```ts import type { ModelMessage } from "ai"; type TrainingExample = | { type: "completion"; prompt: string; completion: string; metadata?: Record; } | { type: "conversation"; messages: ModelMessage[]; metadata?: Record; } | { type: "preference"; input: ModelMessage[]; chosen: ModelMessage; rejected: ModelMessage; metadata?: Record; }; ``` Messages are AI SDK `ModelMessage`s. Use `completion` for prompt/response pairs, `conversation` for multi-turn chats, and `preference` for DPO-style pairs. Anything else about an example (its source, date, author, the session it came from) goes in `metadata`; other top-level fields are an error. Metadata travels with the example through transforms and into eval cases, so any example can be traced back to where it came from. ### What each shape works with A dataset accepts all three shapes, but trainers and evals don't all take all of them. Each trainer checks its data in `start`, before the job is created, and its error names the example to fix: | Shape | `trl()` | `vertex()` | `casesFrom()` for evals | | -------------- | --------------------------------------------- | ----------------------------------- | ----------------------- | | `completion` | yes | yes | yes | | `conversation` | yes, with string content and no tool messages | yes, text only and no tool messages | yes | | `preference` | not yet | no | skipped | `trl()` also needs every example in a dataset to have the same shape. Methods that train on preferences (`dpo`, `preference`) exist in the trainer contract for trainers that support them; neither built-in trainer does yet. See [Local training](https://june.dev/docs/training/local.md#methods-and-artifacts) and [Managed training](https://june.dev/docs/training/managed.md#methods-models-and-options). ## Loading data `dataset()` takes an array of examples, or the path or `file:` URL of a JSONL file with one example per line: ```jsonl {"type":"completion","prompt":"How do I cancel?","completion":"Settings, then Billing, then Cancel plan."} {"prompt":"Where are invoices?","completion":"Under Settings, then Billing."} {"messages":[{"role":"user","content":"Is there a free plan?"},{"role":"assistant","content":"There is."}]} {"prompt":"What's your uptime?","chosen":"99.95% last year.","rejected":"Pretty good I think."} ``` A line without `type` gets one from its fields: `prompt` and `completion`, `messages`, or `chosen` and `rejected`. A preference line may give a `prompt` string instead of `input`, and strings for `chosen` and `rejected`; they become user and assistant messages. Blank lines are skipped. Files are read on first use, relative to the current directory. Every example is checked as it loads, and an error names the file and line: ```text june: ./support.jsonl:12: unexpected field "id" on a completion example; put extra fields in "metadata" ``` An array is copied, and checked when `dataset()` is called. CSV isn't supported yet; convert it to JSONL or map rows into examples in memory. Other parts of June build datasets for you: [`datasetFromSessions()` and `preferenceDatasetFromSessions()`](https://june.dev/docs/chat/learning.md) turn chat sessions and their feedback into conversation and preference datasets, and [`june/copy`](https://june.dev/docs/copy.md) turns a corpus of someone's writing into completion examples. Both return ordinary datasets. ## Reading examples ```ts import { dataset } from "june/train"; const data = dataset("./support.jsonl"); await data.examples(); // a copy of every example, in order await data.size(); for await (const example of data) console.log(example.type); ``` A dataset loads once and keeps its examples, so every call on it sees the same data even if the file changes. Call `dataset()` again to reload. Every call returns copies, and transforms get copies too, so changing an example you were handed never changes the dataset. ## Deterministic splits ```ts import { dataset } from "june/train"; const { train, validation, test } = dataset("./support.jsonl").split({ train: 0.8, validation: 0.1, test: 0.1, seed: 42, }); ``` The shares must add up to 1; leave `validation` or `test` out for an empty split. Each split is a dataset, and together they hold every example exactly once. June ranks every example by a SHA-256 hash of the seed and the example's content, then takes the test split, then validation, from the top of the ranking; the rest is train. So: - The same examples and seed always split the same way, on any machine. - The assignment doesn't depend on input order. Shuffling the file doesn't move examples between splits. - Adding a few examples only moves the examples near the cut-off points, so most of a test set stays put as a dataset grows. - Sizes are rounded from the shares: 8 examples split 80/10/10 give 6, 1 and 1. - Each split keeps the input order of its examples. By default an example is identified by its whole content. When part of an example can change between runs, such as a prompt a model generated, pass `key` with something stable, so the same item always lands in the same split and a re-run can't leak test examples into training: ```ts import { dataset } from "june/train"; dataset("./support.jsonl").split({ train: 0.9, test: 0.1, key: (example) => String(example.metadata?.id), }); ``` ## Fingerprints `fingerprint()` returns `sha256:` over every example in order. Each example is serialized as JSON with sorted keys and without `undefined` fields, metadata included. Key order in the file doesn't matter; anything else does: - Changing an example's content or metadata changes the fingerprint. - Reordering examples changes the fingerprint. June doesn't sort them for you. - Loading the same data from a file or an array gives the same fingerprint. `summary()` returns `{ source, size, fingerprint }`, which `fineTune()` records in its manifest for the train and validation datasets. Two manifests with the same fingerprint were trained on the same examples. ## Provenance A dataset's provenance is its `source`, its fingerprint and its examples' metadata. June records the first two in a [training manifest](https://june.dev/docs/training/fine-tuning.md#the-manifest); it doesn't store datasets, so keep the file a manifest points at, or the examples themselves, wherever you keep artifacts. Record the test split's fingerprint next to your eval results too, so you can show that a model was never trained on what it was evaluated on. ## Transforms Transforms return a new dataset and leave the original alone: | Transform | | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `.map(fn)` | replaces each example with `fn(example, index)`, which may be async; the results are checked | | `.filter(fn)` | keeps the examples `fn(example, index)` returns true for | | `.dedupe(key?)` | keeps the first example for each key; by default the content without `metadata`, so metadata-only differences count as duplicates | | `.split(options)` | see [Deterministic splits](#deterministic-splits) | Each one is added to the dataset's `source`, such as `./support.jsonl | filter | dedupe | split(train)`, so a manifest shows how its data was made.