Lost in Translation: How a Ported Dictionary Inflated Every 'Authority' Score

I spent weeks stress-testing my newest, riskiest metric, and I shipped a full battery of validation sets for it. Meanwhile an older, boring metric — a moral-foundations dictionary — sat untested because it “obviously worked.” It didn’t. A single word, mistranslated across a language boundary, had been inflating one whole moral dimension by about 2× for every speaker, in every text I scored. The bug was not in the algorithm. It was in the dictionary. A ported word list is code — so test it like code.

Scoring morality with a word list

Moral Foundations Theory (Graham, Haidt, Nosek, 2009) says our moral intuitions cluster into a few foundations: care, fairness, loyalty, authority, sanctity. The classic way to measure them in text is a Moral Foundations Dictionary: for each foundation, a list of trigger words. You count how often each foundation’s words appear, normalize by length, and read the profile.

It is deliberately dumb, and that is the point — it is cheap, transparent, and portable to any language if you have a dictionary for it. My pipeline scores French political speech, so it ships a French list:

{
  "care": ["...", "protéger", "souffrance", "..."],
  "fairness": ["justice", "équité", "droit", "égalité", "..."],
  "loyalty": ["nation", "patrie", "fidélité", "..."],
  "authority": [
    "ordre",
    "respect",
    "hiérarchie",
    "discipline",
    "loi",
    "autorité",
    "obéir",
    "...",
    "pouvoir",
    "souverain",
    "..."
  ],
  "sanctity": ["..."]
}

The scorer is as simple as it looks. Lemmatize every token with spaCy, check the lemma against each foundation’s set, normalize per 1000 words:

counts = {key: 0 for key in mfd}
word_count = 0
for token in doc:                       # spaCy Doc over the full text
    if token.is_punct:
        continue
    word_count += 1
    lemma = token.lemma_.lower()
    for foundation, lemmas in mfd.items():
        if lemma in lemmas:             # bag-of-lemmas membership test
            counts[foundation] += 1

scores = {k: round(counts[k] * 1000.0 / word_count, 2) for k in mfd}

Nothing here is wrong. The entire correctness of this metric lives outside the code — in the assumption that each word in the list maps to exactly one sense. That assumption is a property of the dictionary, and it is exactly what a translation can quietly break.

There may not even be an English original to blame. A list like this is often written straight into the target language from an English concept — someone is asked “give me French words for authority” and writes down reasonable words. The translation that gets lost is not word-for-word; it is conceptual. And conceptual ports are the least-audited kind there is.

The smell that saved the metric

I was editing a piece on a farmers’ congress where several politicians spoke one after another — a rare, controlled comparison. The chart for a Green party leader showed authority as her dominant foundation and fairness near the bottom.

Stop and feel how wrong that is. A Green leader whose moral vocabulary is dominated by authority and starved of fairness? Every other signal in the piece said the opposite. The number was not just surprising; it contradicted everything around it.

That contradiction is the whole story. When a metric you trust returns a number you cannot explain, you have two choices: rationalize it (“interesting, she is more institutional than I thought”) or investigate it. I had almost rationalized it into the published piece. Then I pulled the raw list of hits.

More than 30 of her 50 authority hits were not the noun “power.” They were the verb pouvoir — “we can rebuild,” “you could say,” “so that we are able to” — plus a few puis (“then”). The metric was not measuring her relationship to authority. It was measuring how often she said “can.”

The bug: translation-induced polysemy

Here is the mechanism, step by step, because it goes far beyond French or moral foundations.

Step 1 — an entry that feels unambiguous. The authority list needs a word for power, the authority the State holds. The obvious pick is the noun pouvoir — le pouvoir, the thing presidents hold. In the author’s head it has one meaning.

Step 2 — the entry looks correct, and is, in isolation. Le pouvoir is exactly the noun you want. Nobody writing or reviewing the list feels a trap, because on its own the word simply is that noun.

Step 3 — a homograph explodes underneath you. But in French pouvoir is also the modal verb “to be able to,” and one of the most frequent verbs in the language: je peux, il peut, nous pouvons, il pourra, j’ai pu…. A rare noun in the source concept became a top-frequency function word in the target language.

Step 4 — lemmatization collapses it all onto the trigger. This is what makes it lethal instead of merely noisy. A bag-of-lemmas model does not match surface forms; it matches lemmas. And spaCy’s French lemmatizer maps every conjugated form of the modal verb back to the lemma pouvoir:

"peux", "peut", "pouvez", "pouvons", "pu", "pourra", "puisse"  ──lemmatize──▶  "pouvoir"

Worse, it also lemmatizes the common connector puis (“then/next”) onto pouvoir. So even a narrative link counted as authority. The dictionary said “match pouvoir,” and lemmatization dutifully fed it half the modal verbs — and some connectors — in the corpus.

The result is not random noise; it is a structural, correlated inflation. Everyone uses “can / could / able to” constantly, so every authority score was pushed up by a similar amount. The bug was invisible precisely because it was uniform: with everyone inflated, the profiles still looked internally plausible. It only surfaced on a speaker whose true authority was low enough for the noise to dominate — an ecologist. About 60% of every speaker’s authority score was this noise.

One honest detail, and it is itself a small lesson: I checked the fix by running the candidate words through the real spaCy model instead of reasoning about them from memory. That is when I learned that a part-of-speech filter only rescues you for cross-POS homographs. pouvoir is one — the modal verb and the noun sit in different parts of speech, so a POS filter slices them apart cleanly, and that single entry was essentially the whole bug. But words like droit (le droit, “the right/law” vs. tout droit, “straight ahead”) are same-POS: both senses are nouns, so neither the lemmatizer nor the tagger can tell them apart, and no POS gate will help. The clean win is pouvoir; the rest is defensive cladding. You only learn which is which by running the test, not by trusting the comment above the code.

The second bug: a noun-only fairness list

While pulling the thread, the fairness list showed a quieter mistake — same translation mindset, different failure. The French fairness list was built almost entirely from nouns: justice, équité, droit, égalité, injustice, mérite…. It was missing:

• the adjectives people actually speak — juste / injuste, équitable / inéquitable — you say “c’est injuste,” you rarely say “c’est de l’injustice”;
• the distributive-justice vocabulary at the heart of left-wing fairness talk — inégalité, inégalitaire, répartir, répartition, redistribuer, accaparer.

So fairness was systematically under-counted, and worst of all for exactly the speakers who talk about it most. The Green leader again: her real fairness vocabulary was largely invisible to a noun-only list.

Two independent bugs, pushing the same way: authority inflated, fairness deflated. Together they built a consistent, plausible, completely wrong picture — everyone looked more authoritarian and less egalitarian than they were.

The fix

Two changes.

(a) POS-gate the ambiguous lemmas. Keep the entry in the dictionary, but count it only when the token’s part-of-speech matches the moral sense. If the POS is unknown, skip the gate (so non-spaCy callers do not regress):

# Lemmas spaCy collapses across very different senses. Count them only when the
# token's POS matches the moral sense; any other POS is a homograph → skip.
POS_GATED_LEMMAS = {
    "pouvoir": {"NOUN"},          # le pouvoir (authority) ≠ modal verb / "puis"
    "général": {"NOUN"},
    "état":    {"NOUN", "PROPN"},
    "chef":    {"NOUN"},
    "droit":   {"NOUN"},
    "loi":     {"NOUN"},
}

for token in doc:
    if token.is_punct:
        continue
    word_count += 1
    lemma = token.lemma_.lower()

    allowed_pos = POS_GATED_LEMMAS.get(lemma)
    pos = getattr(token, "pos_", "")
    if allowed_pos is not None and pos and pos not in allowed_pos:
        continue   # homograph in a non-moral role (e.g. modal "pouvoir") → drop

    for foundation, lemmas in mfd.items():
        if lemma in lemmas:
            counts[foundation] += 1

(b) Expand the fairness lexicon with the adjectives and distributive-justice terms it was missing.

The effect on the panel: mean authority dropped from about 10 to about 5.5 per 1000 words. The Green leader’s dominant foundation flipped from authority to care; her fairness roughly doubled. Nothing about the method changed — same bag-of-words, same normalization. The dictionary and one POS check were the whole story.

…and the fix quietly repeated the same sin

Here is the part that keeps me honest. Expanding the fairness list, I added the adjective juste — “c’est injuste,” “ce n’est pas juste.” Correct, and necessary. But juste in French is also one of the most common adverbs and filler words in the language: “je veux juste dire,” “c’est juste incroyable,” “juste avant.” Run it through the same lemmatizer and spaCy tags those as ADV, lemma juste. Because I did not gate juste, every filler “juste” now scored as fairness. I fixed an authority over-count and, in the same change, planted a smaller fairness over-count of the exact same species.

I caught it the same way as the first one — by running the sentence through spaCy. The clean remedy is not to gate juste (spaCy calls the good uses ADV too, so a gate would throw them out with the filler) but to drop bare juste and keep the unambiguous injuste / inéquitable / équitable. The lesson could not be more on the nose: the moment you touch the lexicon you are writing code, and the code has bugs — including the one you are in the middle of writing.

The blast radius: a metric that becomes a corpus

Here is what turns a tidy bug-fix into a cautionary tale. A metric like this does not score one text in isolation. Over months it accretes into a corpus — a growing body of published analyses that cross-reference each other (“the highest authority score of any speaker so far,” “more institutional than X”). A biased metric does not produce one wrong number; it produces a wrong coordinate system, and every comparison drawn in it inherits the bias.

So the fix was not “ship the patch.” It was:

• recompute the moral foundations for every already-published piece, from the archived transcripts;
• discover that the dominant foundation flipped for almost half of them (authority → fairness, loyalty, or care);
• rewrite the analysis in each, because “authority dominates everything here” had often been the headline reading — and it was an artifact;
• correct every cross-text superlative (the corpus-wide “authority record” changed hands entirely).

Authority had been seductive precisely because it was inflated: it kept coming out on top, so it kept becoming the story. The bug did not just add noise — it kept nominating itself as the lede.

What the test should have asserted (and now does)

This is the part I owed from the start, and it is the real point of the post. The right shape is three layers, ordered fast to slow — and writing them is what surfaced everything above, including the juste relapse. Every case falls straight out of the bug. The whole battery lives in one file, test_moral_foundations.py.

Layer 1 — code mechanics, no model, microseconds

These pin the scorer. The trick that keeps them instant is that they never load spaCy: they hand the metric small token stubs carrying exactly the two fields the gate reads — a lemma and a part-of-speech tag.

@dataclass
class FakeToken:
    text: str
    lemma_: str = ""
    is_punct: bool = False
    pos_: str = ""

So I can build the precise homograph collision I care about and assert the count, with no fr_core_news_lg in the loop. The test that shuts the relapse from the previous section is four tokens long:

def test_layer1_adverbial_juste_contributes_zero_to_fairness():
    tokens = [
        FakeToken("juste", lemma_="juste", pos_="ADV"),          # "c'est juste" → 0
        FakeToken("juste", lemma_="juste", pos_="ADV"),          # "juste avant" → 0
        FakeToken("injuste", lemma_="injuste", pos_="ADJ"),      # disambiguated → +1
        FakeToken("inégalité", lemma_="inégalité", pos_="NOUN"), # distributive → +1
    ]
    metric = MoralFoundationsMetric(options={"dictionary_file": "data/mfd_fr.json"})
    out = metric.compute(_ctx(FakeDoc(tokens=tokens), REPO_ROOT))

    assert out["raw_counts"]["fairness"] == 2  # injuste + inégalité; both `juste` drop
    # ...and the fix must not add or drop a foundation key downstream code depends on:
    assert set(out["scores"]) == {"care", "fairness", "loyalty", "authority", "sanctity"}

Two adverbial juste score nothing; the two disambiguated terms I kept still score. The last line quietly guards the output shape, so a well-meaning refactor can’t silently rename a foundation out from under the charts.

The sibling test is the one I find most honest, because it asserts a limitation rather than a success:

def test_layer1_pos_gate_is_noop_for_same_pos_homographs():
    # `droit` / `état` / `chef` are tagged NOUN even in their non-moral senses
    # ("tout droit", "en l'état", "chef de gare"), so their POS gate never fires.
    tokens = [
        FakeToken("droit", lemma_="droit", pos_="NOUN"),  # gate {NOUN}       → counts
        FakeToken("état", lemma_="état", pos_="NOUN"),    # gate {NOUN,PROPN} → counts
        FakeToken("chef", lemma_="chef", pos_="NOUN"),    # gate {NOUN}       → counts
    ]
    out = metric.compute(_ctx(FakeDoc(tokens=tokens), REPO_ROOT))

    assert out["raw_counts"]["fairness"] == 1   # droit
    assert out["raw_counts"]["authority"] == 2  # état + chef

This is a canary for a known weakness. The gate cleanly separates pouvoir because its senses straddle a POS boundary (noun vs. verb); it is powerless against droit/état/chef, whose competing senses are both nouns. Rather than pretend otherwise, the test pins the leak — so the day someone makes the gate smarter, they do it on purpose and this red test tells them they changed the contract. (The companion pouvoir test — a modal pouvoir and a puis scoring 0 authority, while le pouvoir + loi + autorité + État score 4 — was already there from the first fix.)

Layer 2 — a data-contract on the shipped dictionary

The lexicon is code — guard it like code. These are the tests I most wish I’d had, because they watch the JSON, which is where the bug actually lived. They read the real file, not a fixture:

REPO_ROOT = Path(__file__).resolve().parents[3]
SHIPPED_MFD_FR = REPO_ROOT / "data" / "mfd_fr.json"

The tripwire is a denylist — the homographs a coarse POS gate can’t save, so the only fix is to keep them out of the file entirely:

# Bare surface forms that must NEVER appear in any foundation of mfd_fr.json.
DROPPED_ADVERBIAL_HOMOGRAPHS = {"juste"}

def test_layer2_shipped_dict_has_no_dropped_adverbial_homographs():
    raw = json.loads(SHIPPED_MFD_FR.read_text(encoding="utf-8"))
    lowered = {f: {w.lower() for w in words} for f, words in raw.items()}
    offenders = {
        f: sorted(DROPPED_ADVERBIAL_HOMOGRAPHS & words)
        for f, words in lowered.items()
        if DROPPED_ADVERBIAL_HOMOGRAPHS & words
    }
    assert not offenders, f"dropped homograph(s) resurfaced in mfd_fr.json: {offenders}"

Re-add bare juste to any foundation and this turns red on the next commit. It is the tripwire for the next pouvoir: when I first seeded the denylist it flagged juste; it stayed red until I dropped the word; now it is green, watching for the next entrant. Its mirror asserts the fix didn’t over-correct — the vocabulary I added has to survive:

def test_layer2_shipped_dict_retains_disambiguated_fairness_terms():
    fairness = {w.lower() for w in json.loads(
        SHIPPED_MFD_FR.read_text(encoding="utf-8"))["fairness"]}
    for term in ("injuste", "équitable", "inéquitable",
                 "inique", "inégalité", "inégalitaire"):
        assert term in fairness
    assert "juste" not in fairness  # the bare adverbial form stays dropped

A third data-contract test catches orphaned gates: every key of POS_GATED_LEMMAS must actually appear somewhere in the dictionary, so a typo’d or dead gate entry fails the build instead of silently protecting nothing.

Layer 3 — an empirical canary through the real spaCy model

Layers 1 and 2 assert what the code does with tags I typed by hand. But the entire disaster hinged on what the model actually tags — and a spaCy upgrade could re-tag puis tomorrow and quietly reopen the wound. So Layer 3 loads fr_core_news_lg and asserts the ground truth the gate is standing on. It is container-only: it skips cleanly when the model is absent (so host CI stays green) and runs inside the image that ships the model.

@pytest.fixture(scope="module")
def fr_nlp():
    spacy = pytest.importorskip("spacy")
    if not spacy.util.is_package("fr_core_news_lg"):
        pytest.skip("fr_core_news_lg absent — Layer-3 canary is container-only")
    return spacy.load("fr_core_news_lg")

def test_layer3_modal_pouvoir_is_verb_not_noun(fr_nlp):
    doc = fr_nlp("Je peux vous dire que nous pouvons y arriver.")
    modal = [t for t in doc if t.lemma_.lower() == "pouvoir"]
    assert all(t.pos_ != "NOUN" for t in modal)  # gated out of authority
    assert any(t.pos_ == "VERB" for t in modal)

def test_layer3_puis_lemmatizes_to_pouvoir_but_is_gated_out(fr_nlp):
    doc = fr_nlp("Nous verrons, puis nous déciderons.")
    puis = [t for t in doc if t.text.lower() == "puis"]
    assert all(t.pos_ != "NOUN" for t in puis)   # never counts toward authority

Two more assert the flip side, so the gate can’t over-fire: le pouvoir in “Le pouvoir doit respecter la loi” really is tagged NOUN (the true positive survives the gate), and juste in “C’est juste, il faut agir” really is tagged ADV — empirical proof that no POS gate could have rescued it, so dropping the bare form was the only fix. That first one is the assertion that would have caught the original incident on day one: it encodes the exact fact — the modal pouvoir is a verb — whose going unchecked is what sent every already-published analysis back for a rewrite.

Layers 1 and 2 run in milliseconds on every commit; Layer 3 pays for the model, once, inside the image. And this is where the moral lands: a single Layer-1 assertion — one modal pouvoir scoring zero authority — would have caught the whole incident on day one; and the Layer-2 denylist is what catches the next one.

The moral: test the boring metric

I want to be precise about what I got wrong, because “write more tests” is too cheap.

I tested the metric I was afraid of, and trusted the one I wasn’t. I had poured validation effort into the new, ambitious metric — labeled sets, baselines, the works. The word list felt safe: it is old, it is textbook, it is just a list, what could go wrong? So it shipped with zero tests. My fear was a bad guide to risk. The ambitious metric was watched; the dumb one was armed.

The danger was in the data, not the algorithm. The scoring code was trivially correct. All the risk had moved into a JSON file that reads like configuration and behaves like code. A ported lexicon is not a resource you load; it is a program whose correctness depends on subtle properties of a language you may not be reasoning about carefully.

Translation breaks the one axiom a bag-of-words model rests on — that one surface form (here, one lemma) maps to one sense. That axiom holds well enough within the language a list was written in, because the author felt the polysemy. Port the list to another language and every entry must be re-audited against the target language’s homographs and against what your lemmatizer does to them. “Correct translation” is necessary and nowhere near sufficient.

So, concretely, if you run a dictionary-based metric across a language boundary:

• Audit every entry for target-language polysemy, especially nouns that are also frequent verbs, adjectives, or function words. Run each one through your actual lemmatizer and look at what collapses onto it.
• POS-gate or disambiguate the ambiguous entries — do not just delete them, or you lose the true positives.
• Keep a golden set of hard negatives — sentences built to trip the homographs — and fail the build if the “obvious” ones ever score.
• Sanity-check profiles against strong priors. An ecologist scoring maximum authority and minimum fairness is not a finding; it is a bug report. The counter-intuitive cell in the table is the cheapest anomaly detector you have.

The metric survived because one score was too absurd to publish. Next time the noise might land somewhere plausible, and there is no smell to follow. That is the whole argument for the test.