Wrong twice about one word: legal polysemy and the excludes I found by reading 31 acts

Brazil’s federal government publishes what it does in the Diário Oficial da União, the official gazette. Tenders, contracts, sanctions and rulings all land there, a few thousand acts a day. RegWatch reads it every morning, matches the day’s acts against the terms each client is watching, and mails a digest. A watch is a set of term groups: all groups have to hit, any term within a group will do.

The previous update was about recall. A Postgres search configuration that did no stemming meant a watch for contrato never saw contratos, and a morning run over 3,338 acts reported zero matches. Fixing that turned the feeds on.

This one is about what came through them. I ran three dummy pilot clients for a week, sat down with the data, and wrote a recommendation that would have deleted every good match in the noisiest feed. Six days later a second look retracted a different recommendation, in the opposite direction.

The watch that worked too well

The client is fictional, an infrastructure firm I invented to have a realistic watch to test against. Its watch is (aviso de licitação | pregão eletrônico | concorrência pública) AND (saneamento | pavimentação): tender announcements about sanitation or paving.

Over six publication days it produced 279 matches, 33 a day. The pavimentação half was good, roughly ten of a twelve-item sample on target. The saneamento half was mostly wrong, and it was wrong at a volume that mattered: the analysis puts one word at about 12% of that client’s total volume.

I still had the act bodies for 200 of the 279, so I read the ones that fired on saneamento. There were 31. About 6 of them are actual sanitation works.

The other 25 fail in two distinct ways, and the first one is a language problem no amount of index tuning reaches.

In Brazilian legal Portuguese, saneamento means curing a procedural defect. Same spelling, same lexeme, entirely different subject. Eleven of the 31 are this: “providências necessárias ao saneamento do certame”, “destina-se ao saneamento e à complementação da instrução do processo administrativo”, “não comportam saneamento na forma do art. 71 da Lei 14.133/2021”. Those are procurement suspensions and audit rulings. A stemmer cannot help. The two senses stem to the same thing, which is the point of a stemmer.

The other fourteen are structural rather than linguistic. Matching is document-level: an act matches if the term appears anywhere in up to 6,000 characters, and that includes the letterhead and the signature block. So an act buying medical supplies matches because a municipal health-and-sanitation secretariat signed it. A state sanitation utility buying household appliances matches for the same reason. None of those acts is about sanitation works. The word is in the name of the body that happened to sign.

The fix I proposed, and why the data killed it

My first recommendation was a title-only scope for the term: only count saneamento when it appears in the act’s title, not anywhere in the body. It reads like an obvious win. A signature block is not a title, so the department-name failures disappear.

Then I checked it against the 31 acts, and it does not survive contact with them. None of the 31 has saneamento in its title. Not the noise, and not the six real ones either. A title scope would have taken the feed from 31 matches to zero and quietly removed the client’s actual results along the way.

The obvious retreat is a softer version of the same idea: if not the title, then position. Noise sits in the letterhead or the signature; signal sits in the body. So I recorded where the word actually fell, as a percentage of the way through each act, for the ten acts where I had that.

Real ones at 45%, 50%, 58%, 63% and 71%. Noise at 6%, 19%, 53%, 85% and 99%. That 53% sits in the middle of the real cluster, so there is no threshold and no “ignore the last 10%” heuristic that keeps the six and drops the rest. Ten acts is a small n. It also only takes one interleaved point to rule a separator out, and there it is.

Position does not carry the signal. Neither does field. What does carry it is the phrase sitting next to the word: saneamento do certame, saúde e saneamento, passível de saneamento. I simulated twelve such phrases against the 31 acts. They remove 17, and every one of the 17 is noise.

The mechanism to do that already shipped. Watches have had an exclude list since early on, and it operates on exactly that level, a literal phrase anywhere in the act. What I wrote in the analysis at the time still reads as the correct diagnosis:

The mechanism was never the problem. The problem is discovery.

Nobody authors twelve phrases like those from a blank page. I found them by reading 31 acts one at a time, which is not a thing a client is ever going to do, and not a thing I want to do again per term per watch.

Finding the phrases without reading the acts

So I built the discovery step. watch_term_contexts --watch <id> walks every act a watch matched, finds each of the watch’s terms in the text, and counts the recurring phrases on either side of it.

The interesting part is that “either side” has to mean two separate counts:

# search_text is already lowercased and accent-stripped, so \w is
# enough for word boundaries here. Left and right context are counted
# SEPARATELY: a two-sided window fragments on the numbers and dates that
# follow most DOU phrases, so the recurring half never aggregates.
reps = "{1," + str(window) + "}"
left_re = re.compile(r"((?:\w+\W+)" + reps + r")" + esc)
right_re = re.compile(esc + r"((?:\W+\w+)" + reps + r")")

My first version used one window spanning both sides, and it returned almost nothing above a count of one. Gazette prose is full of process numbers, dates and article references, so saúde e saneamento is followed by something different in every single act. Pairing the two sides destroys the left side’s repetition. Counting them separately, saude e saneamento appears as one phrase with a count, which is what you can act on.

The other decision that made the output usable is throwing away everything that occurred once. A phrase seen a single time is an anecdote and there are hundreds of them; the report keeps only n > 1. On the noisy watch it printed this:

  term 'saneamento' (concept) — literal in 31 act(s)
    preceded by:
         7  saude e saneamento
         4  municipal de saneamento
         3  parentais e saneamento
         2  necessarias ao saneamento

Two of those four are the failure modes from the census, in the order I would want to fix them, from a single command over acts the watch had already matched. Acts whose text has aged past the retention window are counted and reported as skipped, so a shrinking corpus shows up as a number rather than as a quietly weaker report.

That went in on 2026-08-20 as commit d226d59, alongside two other fixes from the same review.

Six days later, the tool had not been used once

I ran the same review again on 08-26 to see what had moved. The pipeline half was healthy: 300 more matches, every one enriched, sixteen of sixteen digests delivered first try. The product half had not moved at all.

Every one of the five active watches still had exclude = []. The tool built specifically to make those writable had produced output that nobody, meaning me, had turned into a single exclude. Two lines of a management command’s output are not a fix; typing the exclude into the watch is the fix, in the same console I rebuilt around a live demo, and I had shipped the part that was fun.

So I did the boring half. Re-running the report over the larger corpus surfaced saude e ahead of saneamento nine times now instead of seven, and I read all nine of those acts individually: ambulance purchases, medical-hospital supplies, funeral services, cooking gas, medical equipment. Nine out of nine noise, zero real works. In every one, the word appears only inside the name of the signing secretariat. saude e saneamento went into the watch as an exclude the same day, after checking that it drops exactly nine acts and none of them mention pavimentação.

The second retraction went the other way

The same 08-20 review had a second tuning recommendation, for a different fictional client watching energy-regulator acts. Forty-four of its 121 matches, 36%, were one routine act type: sign-offs on telecom companies renting space on power poles. A third of the feed doing nothing for the reader. Drop the hook that catches them.

Over the following four publication days, that act type appears zero times in 51 matches. The feed became individually-numbered rulings with no dominant type. The August concentration was transient.

Had I acted on that recommendation, I would have narrowed a watch against a pattern that stopped the day after I measured it, and I would have had no way to notice, because the thing I removed would then never have appeared again either way.

It is not the first time a number I trusted dissolved once I measured it instead of reasoning about it, after a cheaper model that turned out to cost 51% more. The difference between the two calls is not judgement, it is method. The saneamento finding is a census: all 31 acts, read individually, classified by hand. The homologação finding is a proportion over six days of one feed. One of those survives a second look by construction and the other one was always a sample of a moving thing, and on 08-20 I wrote them into the same numbered list at the same confidence.

The number under all of it

There is a figure in both reviews that makes every other figure in this post softer than it looks. At the second review, 947 of 950 matches were still in state new. The same three had been triaged on 08-20, and no more.

No labels means no precision metric. Every “about 6 are real” and “nine out of nine noise” here is my own read of the text, not a measurement against anyone’s judgement of what they wanted. I can say that with numbers attached because I read the acts, and it is still one person’s opinion sampled at n = 31 and n = 9. The pipeline can tell me exactly how many acts it scanned and matched, to the row. It cannot tell me whether a single one of them was useful, and that is the gap that decides whether this product is worth anything.

It is the same shape as the problem I wrote about building a regress gate for an agent: the machinery that runs the thing is easy to observe, and the machinery that says whether the output was any good has to be built separately and deliberately, or it does not exist.

What I learned

A census and a sample are not the same evidence and should not arrive in the same list. Both of my recommendations were written in the same voice, numbered one after the other, and one of them was a full read of every matching act while the other was a ratio over six days. Six days later one still held and the other had evaporated. The fix is to label a finding census or sample where it is written, so future-me can weight it without re-deriving how it was produced.

The other one is that a tool is not a fix. watch_term_contexts was the correct response to the discovery problem, it works, and for six days it changed nothing at all, because building it felt like progress and typing a three-word phrase into a config field did not. The gap between “we now have the ability to do X” and “X is done” is where a lot of my unfinished work lives, and it does not show up in a commit log, since the commit that builds the tool looks exactly like the commit that fixes the problem.

What’s next

The storage side of that second review turned out to be the urgent one. A weekly cleanup job had a Sunday trigger, and had never produced a single execution, silently, because it was missing one IAM binding that nothing in the system was watching for. The alerting could not have caught it either: every policy triggers on a job that ran and failed, and this job never started. At the growth rate I measured, the database was about six publication days from breaching the 500 MB free tier. That is the next update.

The question I keep turning over: how would you get a first set of relevance labels out of a feed that nobody is being paid to read yet? Triage-in-the-digest, a thumbs up per item in the email, seems obvious and I suspect it gets used twice and then never again. If you have shipped a feedback loop that actually collected labels from real users on a low-stakes feed, I would like to know what it looked like. The series is where I write down the parts I get wrong, and this update had two of them.