A reproducible benchmark that puts numbers on browser-automation stealth
Scores browser-automation stealth configs against self-hosted bot detectors on your own localhost. It measures detectability only, and every detector runs offline.
Update 5 of the stealthbench build log. stealthbench scores browser-automation stealth setups against open, self-hosted bot detectors on your own localhost and commits every result. The last story on the roadmap was packaging, and its spec left one thing deliberately unpinned: publish to real PyPI now, or wire the release pipeline against TestPyPI and stop. This update is that decision and what it cost. I aimed the pipeline at TestPyPI, and the reason is visible inside the wheel itself: `python -m zipfile -l` shows the four vendored detector panels ship, but `node_modules/` doesn't, and `botd.html` imports its detector bundle from exactly there. So the wheel carries a detector page and not the bundle that page loads, on top of needing Chrome, Camoufox's patched Firefox, and Node that no Python package can provide. The release itself uses PyPI's OpenID Connect trusted publishing: `id-token: write` lets the job mint a short-lived identity per run and the index verifies which repo, workflow, and environment it came from, so no long-lived API token is stored anywhere. Two console scripts (`stealthbench`, `stealthbench-serve`) give the module invocations from the last two updates real names, and the version went 0.0.1 to 0.1.0. The honest ending: `git tag` returns nothing, so the trigger has never fired, which was a choice. But checking rather than asserting turned up something I hadn't chosen: `gh api .../environments` returns an empty list, so the `testpypi` environment the workflow names doesn't exist, and the pipeline I'd been calling wired is one step short of it. The only build in existence is the one `uv build` made on my laptop in a git-ignored `dist/`.
Update 4 of the stealthbench build log. stealthbench scores browser-automation stealth setups against self-hosted bot detectors on your own localhost and commits every result. For three updates I'd pitched it as 'anyone can rerun this,' but the actual bring-up was a five-step shell ritual I'd memorized: npm ci and serve the panels on :8901, git clone CreepJS and serve its docs on :8902, then run. This update collapses the four-line server half of that to one command, `python -m stealthbench.serve`, a stdlib-only module (http.server, subprocess, urllib, no new deps) that runs npm ci, fetches CreepJS once, serves both roots, and health-checks each URL until it returns 200 before it says ready. That surfaced a real design question: the whole project rests on 'detectors self-hosted on localhost only,' yet serve does a `git clone` from GitHub. I drew the line at setup versus run time: the fetch is a one-time, idempotent setup step (skipped if the dir exists), and the bench itself still contacts only localhost. The other landed story is `--config`/`--detector` flags to run a subset, built so a bare run is byte-for-byte the old matrix (the no-selection path returns the same list object, identity, so the numbers can't move), a de-selected cell is absent and never a fabricated 0, and an unknown name errors at parse time. The third story, PyPI packaging, I cut.
Update 3 of the stealthbench build log. stealthbench scores browser-automation stealth setups against self-hosted bot detectors on your own localhost and commits every result. This update widens the matrix: two more detectors (a Sannysoft-style panel and rebrowser-bot-detector) and a fifth config, nodriver. nodriver is async and the bench's BrowserHandle contract is sync, so the new handle drives each call to completion on a private event loop. Wiring it in surfaced a real bug: nodriver's tab.evaluate(return_by_value=True) drops falsy results (its unwrap tests `if remote_object.value:`, so 0/false/'' vanish) and leaves plain objects wrapped in a RemoteObject, so detector signals came back as None or garbage. The fix is to stop fighting its unwrap: wrap the script in JSON.stringify and json.loads the returned string, which round-trips every value faithfully. With that fixed, the two new panels mostly corroborate the tells ranking (undetected-chromedriver and nodriver ace both, vanilla and selenium-stealth get dinged), and camoufox runs only 9 of rebrowser's 10 tests because one is Chrome-specific.
Update 2 of the stealthbench build log. stealthbench scores browser-automation stealth setups against self-hosted bot detectors on your own localhost and commits every result. Update 1 reported one number per config; this update ran each config ten times and drew the spread on the chart. Two of the four scores don't repeat: stock Selenium and undetected-chromedriver wobble run to run (undetected-chromedriver even hits a clean 100% on some trials), while selenium-stealth and camoufox land on the same number every time. The bars are descriptive spread (min/max/mean/population stdev), deliberately not a confidence interval. Also in this update: the results schema bumped to v2 while still reading every v1 snapshot, a history layer aggregates committed snapshots into a trend, and run metadata records browser and driver versions from the environment (numbers only, no IP or fingerprint).
Update 1 of the stealthbench build log. stealthbench is a reproducible benchmark that puts numbers on browser-automation stealth: it scores stealth configs (how you set up an automated browser) against self-hosted detectors (how a site decides you're a bot), on your own localhost, and commits every result. This update adds the fleet's first non-Chrome arm, a stealth Firefox (Camoufox driven through Playwright), and does it the way the architecture was built to allow: a new Config plus a new BrowserHandle adapter, with zero changes under detectors/. The BrowserHandle Protocol is the Ports and Adapters seam; a test (test_seam.py) fails the build if a browser SDK is imported outside configs/, and the sprint's proof is an empty `git status src/stealthbench/detectors/`. The one real roadblock was Firefox's Xray wrapper hiding the detector pages' window globals, fixed by reaching through window.wrappedJSObject in the evaluate translation. Camoufox lands at 82% on the tells panel and passes BotD and CreepJS; the three tells it 'fails' are Chrome-specific, so a genuine Firefox doesn't match them.