All Greek to the embeddings: the RAG retrieval floor my own dogfood broke in 43 minutes

I’m building Turmarium in public: a multi-tenant B2B SaaS for schools. A school signs up as a tenant, registers its people and its academic structure, and runs a real term on it. The first six updates built the parts you can’t retrofit, ending with the matrícula and where each rule lives.

In August it grew a module that wasn’t in any of those plans. A teacher uploads their own material, the app parses and embeds it, and a model drafts listas de exercícios grounded in the passages that material actually contains. Every draft carries the page it stands on, and a human accepts or rejects it before a student ever sees it.

I’m the only user of that module so far, which matters for this update, because I also teach physics. So when I dogfood it, the person typing the topic into the box is the same person who wrote the retrieval code, and the two of us disagreed more than I expected.

What the module was missing

Retrieval here is the ordinary shape: embed the topic, ask Postgres for the nearest chunks by cosine distance through pgvector, hand those passages to the model as its only permitted source. Every one of those queries filters organization_id first, with row-level security as the second layer, the way the rest of the app has worked since update 1. I’ve written about the same mechanism as an agent’s long-term memory, and the query is a one-liner either way.

The gap was the nearest. retrieve returned the closest chunks however far away they were, which means a topic the book covers well and a topic the book has nothing on produced the same shaped answer: eight rows, ordered by distance, no complaint. “There is nothing on this topic” and “here are the least-unrelated paragraphs in the book” were indistinguishable to every caller downstream.

The only thing standing between a thin corpus and an invented question was a minimum passage count. That’s a real guard and it’s a blunt one: it catches “almost nothing came back” and it cannot catch “eight paragraphs came back and all of them are about something else.”

So the epic added a distance floor. Which meant I had to pick a number.

Measuring the number

The number is a property of the embedding space, so I refused to guess it. I wrote a management command that prints the per-chunk cosine distances for a list of topics against one organization’s corpus, and ran it against the real thing: a 490-page pt-BR physics textbook, 7,520 chunks, all embedded, gemini-embedding-001 at 1024 dimensions.

Six topics. Three the book demonstrably covers, taken from the drafts I’d already verified by hand. Three I invented to have nothing to do with a physics book: a chocolate cake recipe, Portuguese grammar, the causes of the French Revolution.

The real topics landed between 0.209 and 0.217. The junk landed between 0.332 and 0.399. A gap that clean makes the decision arithmetic, so I took the midpoint, rounded to two places, and wrote it into settings with its provenance attached:

# A2b — the distance floor. MEASURED 2026-08-14 against [the book] (490
# pages, 7,520 chunks) with gemini-embedding-001 at 1024 dims.
# Three known-good topics landed at 0.209-0.217 and three known-junk ones at
# 0.332-0.399, so the floor is the midpoint of a clean gap and not a guess.
RETRIEVAL_MAX_DISTANCE = env.float("RETRIEVAL_MAX_DISTANCE", default=0.27)

The filter itself goes into the SQL rather than into Python, and that placement is the one non-obvious line in the change:

rows = (
    queryset.select_related("document")
    .annotate(distance=CosineDistance("embedding", vector))
    # In SQL, before the slice: filtering `limit` rows in Python would
    # return fewer than `limit` usable hits whenever anything was cut.
    .filter(distance__lte=floor)
    .order_by("distance")[:limit]
)

Filter after the slice and a topic with two far-away neighbours in its top eight silently retrieves six. The floor also hangs off the embedder rather than off the caller, because a cutoff in cosine space describes the space and not the request. The fake embedder used in tests derives its vectors from a SHA-256 hash, so it declares a maximum of 2.0 and no floor at all: thresholding a hash space cuts material for arithmetic reasons that have nothing to do with meaning.

I committed that at 18:10, then moved on to the last task of the epic, which was to sit down and use the thing.

Forty-three minutes

The plan had nominated three dogfood topics, and I picked them precisely because they were the sharp edge: topics written the way I write them on a whiteboard. Two of the three refused outright.

"permissividade do vácuo ε₀ e a lei de Coulomb" retrieved nothing at all. "comprimento de onda λ e frequência" retrieved a single passage, below the minimum. Both refusals are correct behaviour for what the code believed:

if len(passages) < settings.LLM_MIN_PASSAGES:
    # Before the call, not after: refusing costs nothing here, and a model
    # handed two sentences will write a plausible question anyway.
    raise ThinContext(
        f"{len(passages)} passage(s) found for {topic!r}; "
        f"{settings.LLM_MIN_PASSAGES} are required"
    )

Both topics are in the book. Not marginally, not in a footnote: Coulomb’s law has its own numbered section, with a heading that names it. A corpus of 7,520 chunks was telling me it had never heard of either.

I turned the floor off and looked at the real distances. The Coulomb material runs from 0.2701 to 0.2969, so the very nearest chunk of it is past a floor of 0.27 by one ten-thousandth and the hardest one to reach, the section heading that names the law, is past it by 0.027. Every one of those chunks had been sitting there the whole time, on the wrong side of a line I’d drawn 43 minutes earlier.

The reason crossed a module boundary

Symbol-dense queries score worse against this corpus, and the cause is in a module that has nothing to do with retrieval.

Pulling structured data out of an unstructured document is never lossless. When the PDF was parsed, the extractor flattened Greek letters into their Latin lookalikes: σ came out as s, λ as l, and the Coulomb constant’s 1/4πε₀ came out as a small pile of punctuation. The documents lost the Greek. The query kept it, because I typed it. So the two embeddings sit further apart than the physics warrants, and the topics most affected are exactly the ones a physics teacher is most likely to type.

That flattening was already known. I had ruled on it that same morning, at 09:07, and accepted it with no code change: the model infers the intended letter from physics context when it generates LaTeX, which is how I’ve always worked, and the curation screen puts a human in front of the source page anyway. The ruling turned an accident into a contract, and the producer prompt now says out loud that source text may hold Greek transliterated into Latin lookalikes.

Every word of that reasoning is about the author’s eye. Retrieval has no eye: the chunk is selected or it isn’t, hours before any human or model reads a candidate, and nothing in the morning’s ruling covered that path. Turmarium’s import boundaries keep materials from knowing anything about the AI module, which is the right dependency direction and also the reason a decision taken inside the parser could cost 0.05 of cosine distance in a module that never imports it. The finding went into the report under its own heading, with the only note that felt honest: this was not on anyone’s list.

Re-measuring, with the floor switched off

I widened the sample to fourteen topics, ran them with RETRIEVAL_MAX_DISTANCE=2.0 so nothing was truncated, and drew every measured best-distance on one axis.

Nine topics the book covers, five written to be junk. One of the five wasn’t junk: the human circulatory system landed at 0.3266 because the book has a physiology sidebar on electric currents in the heart, so it comes out of the junk set rather than quietly dragging the boundary down. That leaves the worst real topic at 0.2701 and the nearest genuine junk at 0.3318. Midpoint, rounded: 0.30. All three dogfood topics compose again.

Two things on that plot are worth more than the number.

The first is that the symbol penalty isn’t uniform. Two of the three symbol-dense topics are the two worst real topics measured, which is the finding. The third, the surface charge density σ, sits comfortably inside the plain-language cluster at 0.2243. Whatever the parser did to that page, it didn’t cost the same everywhere, and a rule that says “Greek costs you 0.05” would be an overreading of three points.

The second is where the junk actually landed. Every junk hit is a heading or a fragment: EXERCÍCIOS, PROBLEMAS, e, portanto,. Short, structurally empty chunks are attractors for off-topic queries, because they carry almost no signal to disagree with. The gap between the bands looks clean partly because the junk side is being held up by boilerplate. Filter those chunks out and the junk minimum moves further away, which would make the floor safer than it currently is.

The margins, and which way to be wrong

0.30 sits 0.030 above the worst real topic and 0.032 below the nearest junk, which sounds comfortable until you measure against chunks instead of topics. The hardest real chunk to reach, that same Coulomb heading, is at 0.2969. The margin there is 0.003, and I wrote that into the settings comment rather than the report, because the comment is what the next person reads:

# The bands nearly touch, and the margin above the worst real chunk is ~0.003.
# If this needs to move again, move it UP: a floor that cuts real material turns
# a good corpus into `thin_context` and blocks the author entirely, whereas a
# floor that lets junk through is caught by the curation screen this epic
# shipped. Re-measure with `manage.py retrieval_distances` before touching it.

The two failure directions aren’t symmetric, and that asymmetry is the part that transfers to any similarity threshold. Too tight and a good corpus reports itself empty: loud, immediate, and it stops the work. Too loose and a weak passage reaches a draft, where the screen this epic exists to provide catches it. Given a threshold you can only calibrate approximately, err toward the failure a human is already positioned to catch.

I’d hold that argument a little less proudly after what came next, but it’s still how I’d set the number.

The lesson was about the sample, not the arithmetic

The first floor wasn’t miscalculated. Both floors are the midpoint of a measured gap, computed the same way; the second one is just the midpoint of a gap I hadn’t finished measuring. The three real topics behind the first one were all phrased the way I type into a test.

A floor calibrated on topics written by the person calibrating it measures how that person writes, not what the corpus contains.

Calibration inputs are a test set, and mine had one author. It’s the same failure I’ve written about when building eval harnesses for agents: the set you assert against decides what you can discover, and a set drawn from your own habits can only confirm them. Fourteen topics is still a small sample. It’s a small sample with two kinds of phrasing in it now, which is the part that made it useful.

Five days later, the same knob wasn’t the lever

On the 19th, composing starved again. A request for eight questions on Coulomb’s law came back with grounding that contained neither the equation nor the electrostatic constant, and one draft cited a passage while stating a constant that passage doesn’t contain: a citation wider than the source it points at, which is precisely the failure the human screen is worst at catching, because it looks correct and comes with a receipt.

Having just been burned by a threshold, my hand went straight to the threshold. So I measured first, on one chapter of the same book, 516 chunks.

The equation chunk passes the floor at 0.2954. It was being cut by limit, which was still 8 for a request of eight questions exactly as it was for a request of one, so the whole ask reached 28 chunks, about 5.4% of the chapter. A chunk of undelimited LaTeX ranked second at 0.2482, which also half-falsifies my assumption that the parser’s mathematics was uniformly far away. Widening the budget to 24 hits and 8 passages reached 54 chunks, 10.5%, containing both the equation and the constant.

RETRIEVAL_MAX_DISTANCE was not touched. The fix was a retrieval budget that scales with the size of the ask, and the floor stayed at 0.30, where it still is.

What’s next

The parser’s Greek is still flattened, by an active decision rather than by neglect, and fixing it would tighten the whole distance band and let the floor be set with a real margin instead of 0.003. That sits in the backlog next to the other half of the same ticket: filtering the structurally empty chunks that hold the junk band up. Both would move the number, and both would need the measurement run again from scratch, since a re-ingest at a different chunk granularity invalidates every figure in this post.

The question I’d put to anyone shipping a similarity cutoff: where did your calibration inputs come from? Mine came from my own hands, and they measured the wrong thing for 43 minutes. I’d like to know whether anyone has found a discipline that catches this before the dogfood does.