A release with no secrets: OIDC trusted publishing, and the tag I didn't push

stealthbench scores browser-automation stealth setups (configs) against open, self-hosted bot detectors on my own localhost, and commits every result. It’s a benchmark you run, not a library you import.

The last story on its roadmap was packaging, and the spec left one thing deliberately unpinned: publish to real PyPI now, or wire the release pipeline against TestPyPI and stop at that. I’d already cut packaging once, arguing that pip install stealthbench would hand you a command that can’t actually run. So this update isn’t the packaging work. It’s the decision, and a thing I found in the wheel that settled it.

The fork: the real index or the practice one

Publishing to PyPI proper is a one-way door in a way tags and branches are not. The name is claimed, the version is permanent, and pip install stealthbench starts working for strangers who will reasonably assume it does something. TestPyPI is the same protocol against a throwaway index: the whole pipeline runs for real, and nobody stumbles into the result from a search.

The argument for going straight to real PyPI was that the pipeline is only proven when it publishes somewhere that matters. The argument against was that I’d be putting a package on the index everyone searches specifically to prove a workflow turns over, which is using other people’s attention as my test fixture. I took TestPyPI. The machinery is real either way, and the cost of being wrong is much lower on the index nobody browses.

What’s actually inside the wheel

Then I looked in the build, which I should have done before arguing about any of this:

$ python -m zipfile -l dist/stealthbench-0.1.0-py3-none-any.whl
stealthbench/detectors/assets/botd.html                     914
stealthbench/detectors/assets/package.json                  303
stealthbench/detectors/assets/rebrowser.html               2772
stealthbench/detectors/assets/sannysoft.html               1227
stealthbench/detectors/assets/tells.html                   3078
stealthbench/detectors/assets/rebrowser/index.js          17155
stealthbench/detectors/assets/sannysoft/sannysoft.checks.js 6378
...

The vendored panels ship. All four of the self-hosted detectors I built up over update 3 are sitting right there in the wheel, licenses included. What doesn’t ship is node_modules/, because it’s git-ignored and has no business in a distribution.

Now read the first line of botd.html:

<script type="module">
  import { load } from './node_modules/@fingerprintjs/botd/dist/botd.esm.js';

The wheel carries the BotD page and not the bundle that page imports. That’s not a packaging bug I need to fix, since the bundle arrives via npm ci when stealthbench-serve brings the detectors up. But it is the cleanest possible statement of the thing I’ve been asserting in prose for two updates. Even ignoring Chrome, Camoufox’s patched Firefox, and Node, which no Python package can install for you, the wheel’s own detector page has a dangling import until something outside the wheel resolves it.

So the README says so, in the words that shipped:

pip install ships the runner + CLI only. pipx install --index-url https://test.pypi.org/simple/ stealthbench (TestPyPI) gives you the stealthbench and stealthbench-serve commands — but a real run still needs a browser stack (Chrome/Chromium, Camoufox/Firefox, Node.js) and the detector servers (stealthbench-serve). Install ≠ ready-to-run.

A release with no token in it

Here the sprint got more interesting than I expected. The usual way to publish from CI is to generate a PyPI API token, paste it into repository secrets, and hope it never leaks and that you remember to rotate it. I didn’t want a long-lived credential in the settings of a project whose entire pitch is that it talks to nothing but localhost.

The pattern that removes it is OpenID Connect trusted publishing, and it’s worth spelling out because “no token” sounds like hand-waving until you see where the trust actually lives:

1. You register a pending publisher on the index once, naming the owner, the repository, the workflow filename, and the GitHub environment allowed to publish.
2. On a run, the workflow asks GitHub for an OIDC identity token. That’s what permissions: id-token: write grants, and it’s scoped to that single run.
3. The token is a signed claim about provenance: this repo, this workflow, this environment, this ref. It’s short-lived and useless anywhere else.
4. The publish action hands it to the index, which checks the claims against what you registered and returns a one-time upload credential.

Nothing long-lived is stored on either side. The credential exists for the length of one upload, and its authority comes from where the run happened rather than from a string somebody has to keep safe.

In the workflow that collapses to a few load-bearing lines (abridged: the real file also has runs-on, a checkout, and a setup-uv step):

on:
  push:
    tags: ['v*']             # a tag is the only trigger

jobs:
  build-and-publish:
    environment: testpypi
    permissions:
      id-token: write        # mint an OIDC identity per run, no stored token
    steps:
      # ... checkout + setup-uv
      - run: uv build        # clean sdist + wheel
      - uses: pypa/gh-action-pypi-publish@release/v1
        with:
          repository-url: https://test.pypi.org/legacy/

There’s no password: field and no secrets.PYPI_TOKEN, so there’s nothing to leak in a log or forget in a rotation I’d inevitably skip. A secret you never store is one you can’t spill.

That’s the theory. Then I went and checked my own work:

$ gh api repos/bessavagner/stealthbench/environments
{"total_count":0,"environments":[]}

$ gh secret list          # empty, which is the one result I wanted
$ gh run list --workflow=release.yml   # empty

The workflow names environment: testpypi, and no such environment exists in the repo settings. I’d written one end of a handshake, described the other end in a spec, and never made them shake hands. The step-1 pending publisher isn’t something I can prove I registered either, and the emptiness of everything else is not encouraging on that front.

Two commands get real names

The smaller change: pyproject.toml now declares console scripts, so the python -m invocations from the last two updates become plain commands.

[project.scripts]
stealthbench = "stealthbench.__main__:main"
stealthbench-serve = "stealthbench.serve:main"

stealthbench is the bench runner and stealthbench-serve is the one-command detector bring-up from update 4. The version went 0.0.1 to 0.1.0, which is a normal minor bump for new surface area with no measured value moving. uv build turns the tree into a clean sdist and wheel, and dist/ is git-ignored, while results/ stays committed so every published figure still traces to a file I can point you at. A wheel rebuilds from the source tree whenever I want one.

The button I wired, and the wiring I hadn’t finished

git tag returns nothing, so the trigger has never fired. That part I chose. The empty environment list is a different thing: not restraint, just a step I’d been carrying in my head as done. I caught it because writing this section required checking, and checking is cheaper than asserting.

So the sprint landed console scripts, a wheel that builds clean, a README that warns you off, and a release pipeline that is genuinely tokenless and genuinely unfinished. The only build in existence is the one uv build produced on my laptop, in that ignored dist/. Nothing has been published, not even to the practice index.

I’m leaving it there for now, because deferring a tag costs nothing and undoing one costs a lot: a version number on an index is the only artifact in this project I can’t regenerate, unlike every snapshot in results/. What changed is the next line on the list. It used to read “press the button.” It reads “finish the wiring, then press the button.”

What’s next

Which brings me back to the same gate I’ve been carrying since update 2: a headful CI run, so the snapshot regenerates on a schedule instead of when I remember to. The release workflow now sits in the same directory as the CI it will eventually share a runner with. When a clean-room job can bring the whole stack up and produce a real number, the package will have earned its tag, and so will the number.

If you publish Python packages: have you moved to trusted publishing yet, or is there still an API token sitting in your repo secrets? I’m sold on the no-token release. What I’m genuinely unsure about is the fork above, whether wiring it against TestPyPI and stopping is discipline or just procrastination with a rationale attached. The repo is at github.com/bessavagner/stealthbench.