Texts Everywhere: The Corpus Library I Should Have Built First

I run AFK, a news-analysis side project, and I keep several custom metrics running against French political speech — a Latour-inspired axis projection, a moral-foundations scorer, a topic radar — each calibrated on its own isolated corpus. For a long time, each metric fetched its own text and stored it its own way. Six pools, three formats, zero way to ask one question across all of them.

This is the tutorial I wish I’d had: how to build a living text library in Python — one folder per document, a Pydantic schema, a Parquet index — with no database, and how to query it and carve it into collections once it exists.

Why not a database

I already run Postgres and Qdrant for the rest of AFK. Either was five minutes away. I didn’t use them, for one reason:

A database is a second copy of the truth. The moment you register documents in it, you own a synchronization problem forever — a text edited on disk and never re-registered, a row deleted without its folder.

The queries I actually run are faceted filters — “every oral speech, campaign register, open licence, stratified by party” — not similarity search. That’s WHERE … AND … AND …, and SQL over a rebuilt Parquet file is SQL: DuckDB, pandas and pyarrow all read it natively, so nothing has to ship a query engine.

The rule: the disk is the truth, everything else is derived. One folder per document, validated by a closed schema. The index is a Parquet file rebuilt from those folders. If the index and the disk disagree, the index is wrong — delete it and rebuild it.

One folder, one schema

Each document is <slug>/text.txt (raw prose) plus <slug>/metadata.yml, validated on read by a Pydantic model:

Licence = Literal[
    "public-domain", "cc-by", "cc-by-sa", "cc-by-nc", "cc0",
    "copyright-fair-use", "copyright-unknown",
]

class DocumentMetadata(BaseModel):
    model_config = ConfigDict(extra="ignore")
    id: str = Field(pattern=r"^[a-z0-9][a-z0-9\-]*$")
    title: str
    author: str
    date: Optional[date] = None
    language: str = "fr"
    license: Licence

No database row is more trustworthy than a file a human can open, git diff, and edit by hand. extra="ignore" matters: it means the schema is the contract, not whatever keys happened to be lying around.

Folders in, one Parquet file out

A single script walks the folders, validates every one, and writes one file:

def iter_documents(base: Path):
    for entry in sorted((base / "documents").iterdir()):
        meta, text = entry / "metadata.yml", entry / "text.txt"
        if meta.exists() and text.exists():
            yield meta, text

def load_row(meta_file: Path, text_file: Path) -> DocumentRow:
    meta = DocumentMetadata.model_validate(yaml.safe_load(meta_file.read_text()))
    text = text_file.read_text(encoding="utf-8").strip()
    return DocumentRow(
        id=meta.id, text=text, title=meta.title, author=meta.author,
        date=meta.date.isoformat() if meta.date else None,
        language=meta.language, license=meta.license,
        word_count=len(text.split()), char_count=len(text),
    )

table = pa.Table.from_pylist([r.model_dump() for r in rows])
pq.write_table(table, out_path, compression="snappy")

python -m corpus_builder.build runs this over every folder and writes build/reference.parquet. The gate is binary: if one document fails validation, nothing is written — no partially-built index that looks fine until you query the missing tenth of it.

Identity: a filename is not an identity

The first version of my library kept one flat id per document. It broke the day two pools each held a legitimate derivation of the same interview — full Q&A vs. interviewee-only, 10,570 words vs. 7,503. Same source, two documents, one collision. Worse: the writer treated an existing folder as “skipped (exists)” — silent data loss dressed up as success.

The fix is to identify a document by (source_id, variant), never by id alone:

source_id: Optional[str] = Field(default=None, pattern=r"^[a-z0-9][a-z0-9\-_]*$")
variant: str = Field(default="raw", pattern=r"^[a-z0-9][a-z0-9\-_]*$")
derived_from: Optional[str] = None   # parent id — required once variant != "raw"
derivation: Optional[str] = None     # the cleaner/operation that produced it

@property
def group_key(self) -> str:
    return self.source_id or self.id   # the leak-guard key

@property
def doc_key(self) -> tuple[str, str]:
    return (self.group_key, self.variant)

derived_from is written at derivation time by the code that produces it, never re-inferred from a title hash — I lost a parent link once to a single closing quote eaten during cleaning, and that was luck, not design.

The writer now reports what actually happened on disk, instead of a single ambiguous “skipped”:

created — the folder didn’t exist.
updated — same identity, same prose, different metadata.yml → rewritten. A migration that fixes a licence or a derived field is the normal shape of this branch.
unchanged — byte-identical prose and metadata: the real no-op.
collision — same identity, different text → raises. Never overwritten, never skipped quietly.

Building the index

The library’s index is declared with an explicit Arrow schema — not inferred from the data:

INDEX_SCHEMA = pa.schema([
    ("id", pa.string()),
    ("source_id", pa.string()),
    ("variant", pa.string()),
    ("register", pa.string()),
    ("license", pa.string()),
    ("redistribution", pa.string()),
    ("collections", pa.list_(pa.string())),
    ("government", pa.string()),
    ("theme", pa.string()),          # reserved for the annotation layer, see below
    ("publishable", pa.bool_()),
    ("word_count", pa.int64()),
    # …
])

Why bother, when pyarrow can infer types? Because an all-empty column like theme would otherwise land as type null and silently change type the day a single value appears. A schema that depends on the data isn’t a contract. With an explicit one, two rebuilds of the same folders are byte-identical, which turns “the index is a pure function of the disk” into an actual check instead of a hope:

rm -rf index/ && python library.py reindex     # byte-for-byte identical Parquet
python library.py reindex --check              # the same check, writing nothing

reindex also refuses to write when a document is invalid or two documents claim one identity — an index silently missing a document is exactly the drift this layer exists to prevent.

Once index/index.parquet exists, querying it is filtering a list of dicts (or a DuckDB SELECT, if you’d rather):

python library.py query --register interview --medium written --redistribution allowed
python library.py query --collection moral-reference --format table
python library.py query --where theme=defense --format manifest

A few rules earned their place the hard way:

One variant per source_id by default. Two derivations of one text must never land in different splits of a gold set — so a query collapses them, keeping raw when present, and says on stderr how many it folded. --all-variants opts out explicitly.
Sort by the column’s own type. A numeric column compared as text orders 1216 before 320; a --limited roster would silently become the wrong roster.
Booleans are parsed strictly. Mapping “anything that isn’t true” to false would make --where publishable=allowed — a plausible typo for --where redistribution=allowed — return exactly the documents that must not ship, and exit 0 about it.

Collections, without a database of collections

Project membership is a field, never a directory:

collections: [moral-reference]

docs/ stays one flat namespace; a document can belong to zero, one, or several collections, and --collection moral-reference is just another facet on the same query. No junction table, no second source of truth to keep in sync with the folders.

Facets a human curates (author, licence, register, collections) live in metadata.yml. Facets a model computes never touch that file — they’re written to a stamped sidecar instead:

{"id": "2026_lfi_melenchon_meeting-saint-denis", "value": "defense",
 "method": "seed-cosine", "engine_version": "cap-fr-theme-grid@v1",
 "computed_at": "2026-07-31T10:00:00+00:00"}

A row missing any of those three stamps fails the read — an annotation whose provenance is unknown is worse than a missing one, since it still answers queries. Recalibrating an engine rewrites that one file; it never opens ten thousand metadata.yml files to do it.

What this bought me

Today the library holds a few hundred documents, rebuilt in under a second, zero identity collisions. Adding a text is one command; querying it by any facet is another. The hardcoded source rosters that used to live duplicated across two Python files are gone — they’re a query now.

The Latourometre calibration corpus and a much larger unannotated reserve are still deliberately outside this library — different constraints, migrated later, a story for another post. The part worth keeping from this one is the sequencing: the moment a second metric asks for text, stop and build the library — before the folders, the formats, and the silent collisions pile up on their own.