# Corpora and authorship A corpus is the person's writing, and everything a voice knows about them comes from it. What you leave out matters more than how much you put in: a voice imitates every sample it's shown, so one piece someone else wrote teaches it to sound like someone else. Optimize for writing that is surely theirs before you optimize for size. ## Authorship hygiene Before preparing, remove: - **other people's words:** quoted text, testimonials, press quotes and excerpts from third parties; - **writing with unclear ownership:** collaborative copy, ghostwritten pieces, and anything heavily edited by someone else; - **machine-generated text presented as theirs:** drafts a model wrote that they published with light edits; - **thin boilerplate:** legal footers, autogenerated emails, and lists, tables of contents or reference pages with little prose; - **duplicates**, such as a post cross-posted to several places; - **pieces they've disowned**, writing the person now considers bad or outdated. You can remove samples before they reach the corpus, or mark them and let preparation decide. `prepareVoice()` drops samples with `authored: false` by default, and its [`include`](https://june.dev/docs/copy/preparation.md#exclusion) option takes your own rule, such as also dropping `edited: true`. Marking is better when you want a record of what was left out and why; dropping at collection is better for text that shouldn't be stored at all. Get the person's consent first. A voice is a model of a real person's writing, and preparing and writing send their text to the models you choose; see [corpora and consent](https://june.dev/docs/reference/security.md#training-on-what-people-wrote). ## Building a corpus from reviewed sources This is the recommended way to build a real corpus. For anything bigger than a handful of samples, keep the decisions in a reviewed list rather than in code that scrapes. June.dev builds its author's corpus this way, and the shape works for any person or brand. Each entry says where a piece lives, which revision to read, what kind of writing it is, who wrote it, and whether it's in: ```ts interface WritingSource { id: string; /** Where to read it: a URL, or a repository and path. */ repo: string; path: string; /** A pinned revision, never a moving branch, so the corpus is reproducible. */ ref: string; /** From a small, fixed vocabulary, since retrieval matches on it. */ type: "essay" | "tutorial" | "short-post" | "readme" | "docs" | "page"; date?: string; authorship: "theirs" | "mostly-theirs" | "uncertain" | "other"; include: boolean; /** Why it's excluded, or anything a reviewer should know. */ reason?: string; } ``` Then a small script reads each included source at its revision, extracts the prose (dropping code blocks, markup, images and blockquotes), turns it into a sample whose metadata carries the entry's `type`, `date` and provenance, and writes the corpus file together with a summary of what was excluded and why. A few rules make this trustworthy: - **Pin revisions.** Reading at a commit or snapshot means later edits, by anyone or anything, can't leak in, and the same list always builds the same corpus. - **When in doubt, leave it out.** A smaller corpus of writing that is surely theirs beats a bigger one that isn't. Exclude `uncertain` and `other` until the person confirms. - **Fail whole.** If a source can't be read, don't write a corpus without it: a corpus missing a source is silently a different corpus. - **Write the exclusions down.** "84 words of prose once code is removed" or "a list of videos, not prose" is the review a person needs, and the reason a sample isn't there later. The script's output is a JSONL file in the [format below](#input-formats). ## Input formats `copyCorpus()` takes an array of samples, or the path or `file:` URL of a JSONL file: ```ts import { copyCorpus } from "june/copy"; const corpus = copyCorpus([ { text: "Recruiting, minus the recruiting.\nFonzie does the sourcing so you can do the building.", metadata: { date: "2025-03-14", source: "homepage", type: "homepage-hero", audience: "startup founders", }, }, ]); ``` A JSONL file has one sample per line, each a JSON object with `text` and optional `id` and `metadata`. Blank lines are skipped: ```jsonl {"id":"launch-tweet","text":"Fonzie now books interviews for you. Pick a slot, we handle the rest.","metadata":{"date":"2026-08-20","source":"twitter","type":"tweet"}} {"id":"changelog-ats","text":"Greenhouse sync is here. Candidates you like in Fonzie show up in Greenhouse, notes and all.","metadata":{"date":"2026-07-01","source":"changelog","type":"changelog"}} ``` A file is read on first use, relative to the current directory, and kept: later calls see the same samples even if the file changes. Call `copyCorpus()` again to reload. Those are the only two formats. Anything else, such as a folder of Markdown posts, a CMS export or a mailbox, is ingestion your app does: read it, decide what's a sample, and hand `copyCorpus()` an array. The Copy Agent, for example, reads a folder of `.md` and `.txt` files, one sample per file, with `key: value` front matter as metadata. ## Samples and ids **Text is kept byte for byte.** June never trims, rewrites or normalizes it, and every completion it later trains on is this exact text. Strip what isn't the person's prose (code, quotes, navigation, signatures) before it reaches the corpus. A sample's `id` defaults to a hash of its text, so it's stable across runs. Give your own ids when you have them, such as a post's slug: they make preparation reports, exemplars and training examples easy to trace. Two samples with the same explicit `id` are an error. Two with the same text and no `id` get `sample-` and `sample--2`, and [preparation](https://june.dev/docs/copy/preparation.md#deduplication) drops the second as a duplicate. ## Metadata Every metadata field is optional: | Field | | | ------------------------------------- | ----------------------------------------------------------------- | | `date` | when it was written; a date or anything `new Date()` reads | | `source` | where it came from: `"blog"`, `"email"`, a URL | | `type` | what kind of copy: `"homepage-hero"`, `"tweet"`, `"changelog"`, … | | `audience`, `intent`, `topic`, `tone` | short free-text labels | | `authored` | `false` when the person didn't write it | | `edited` | `true` when someone else edited it | Dates become ISO strings, and `source`, `type`, `audience`, `intent`, `topic` and `tone` are trimmed. Any other fields are kept as they are. Metadata isn't bookkeeping; it's the leverage you have over what a voice does. [Retrieval](https://june.dev/docs/copy/retrieval.md) picks exemplars by comparing a request's labels with each sample's, `type` most of all, so a corpus with a small, consistent vocabulary of types (`essay`, `tweet`, `changelog`) retrieves far better than one where every sample has its own. The same labels let you count samples by kind, which is how you know what forms the voice can imitate. Label what you know; [preparation](https://june.dev/docs/copy/preparation.md#metadata-inference) can infer the rest. ## Provenance Provenance is part of a voice's quality, not just its compliance: when a draft reads wrong, the exemplars are usually why, and you can only fix a sample you can find. Keep enough in each sample's metadata to find the original again: a URL, or a repository, path and revision. June carries metadata through preparation into every training example, so any exemplar or training example traces back to the writing it came from, and a tuned voice's manifest records the datasets it was trained on: ```json { "id": "post-shipping-small", "text": "Ship the smallest thing that teaches you something…", "metadata": { "date": "2024-02-11", "source": "https://example.com/posts/shipping-small", "type": "essay", "authored": true, "provenance": { "repo": "example/site", "path": "posts/shipping-small.md", "ref": "4b1d0c3e…" } } } ``` A corpus's `fingerprint()` is a SHA-256 over every sample's id, text and metadata, in order. A voice records it, so a saved voice [refuses to load](https://june.dev/docs/copy/training.md#loading-a-voice) against a corpus that changed. `source` is the file path, or `"memory"` for an array, and `size()` counts samples. ## Validation Every sample is checked when the corpus loads, and an error names where it is: ```text june: ./fonzie.jsonl:12: "metadata.date" isn't a date ``` A sample needs a non-empty `text` string; `metadata` must be an object, `id` a string, and `date` something `new Date()` reads. Errors from an array name the sample's index (`memory, sample 3`). ## Voice drift People's writing changes. A blog post from 2014 and last month's launch email may not sound like the same person. Give samples a `date`, and decide how old writing should count: - **Leave it out.** `include: (sample) => sample.metadata.authored !== false && (sample.metadata.date ?? "") >= "2023"` in [`prepareVoice()`](https://june.dev/docs/copy/preparation.md#exclusion) (`include` replaces the default rule, so keep the `authored` check), or filter before building the corpus. - **Weigh recent writing more.** [Retrieval](https://june.dev/docs/copy/retrieval.md#recency) already prefers newer samples, a little, and you can raise its `recency` weight. - **Mark eras.** Extra metadata such as `era: "early"` is kept, so a custom retriever or filter can use it. Dates are also how ties break in retrieval, so a dated corpus is more predictable than an undated one. Drift is maintenance, not a one-time choice. Rebuild the corpus when the person's writing has moved on (a new role, a new product, a year of new posts), prepare it again, and [evaluate](https://june.dev/docs/copy/scoring.md#evaluating-a-voice) the new voice against the old one on the same held-out writing before you switch.