<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xml:base="https://docs.spec-kitty.ai/development/testing/testing-parallel/">
  <id>https://docs.spec-kitty.ai/development/testing/testing-parallel/</id>
  <title>Running the test suite in parallel</title>
  <updated>2026-11-02T00:00:00.000Z</updated>
  <author><name>Spec Kitty</name></author>
  <link rel="alternate" type="text/html" href="https://docs.spec-kitty.ai/development/testing/testing-parallel/"/>
  <summary>How to run the Spec Kitty test suite in parallel locally and in CI: the one correct command, why it is shaped that way, and reproducing the coverage-neutrality gates.</summary>
  <content type="html">&lt;span id=&quot;running-the-test-suite-in-parallel&quot;&gt;&lt;/span&gt;
&lt;p&gt;The Spec Kitty test suite runs safely in parallel locally and in CI, typically
at least 2× faster on a machine with four or more cores. This page explains the
one correct local command, why it is shaped the way it is, and how to reproduce
the coverage-neutrality gates CI uses.&lt;/p&gt;
&lt;p&gt;For what to do when a test goes red on CI &lt;em&gt;unrelated to your diff&lt;/em&gt; — budget gates
vs. correctness flakes vs. environmental flakes, and why we never retry-to-green —
see the &lt;a href=&quot;/development/testing/testing-flakiness/&quot;&gt;test-flakiness handling policy&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;the-local-command&quot;&gt;The local command&lt;/h2&gt;
&lt;p&gt;Do not hand-roll a broad pytest invocation — &lt;code dir=&quot;auto&quot;&gt;make test-fast&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;make test-full&lt;/code&gt;
already encode the parallel/serial split below, and a hand-rolled command drifts
out of date the moment that split changes:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;make&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;test-fast&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt;    &lt;/span&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# fast tier of the typical blast-radius directories (target &amp;#x3C;2 min)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;make&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;test-full&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt;    &lt;/span&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# everything, parallel + serial passes — CI&apos;s target, not PR validation&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;make test-full&lt;/code&gt; runs the bulk of the suite across worker processes, then gives
each of the two parallel-unsafe families (&lt;code dir=&quot;auto&quot;&gt;stress&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;timing&lt;/code&gt;) its own
dedicated serial pass — see &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/blob/main/AGENTS.md#commands&quot;&gt;AGENTS.md&lt;/a&gt; for the
exact pass breakdown and the PR-validation test policy.&lt;/p&gt;
&lt;h2 id=&quot;why---dist-loadfile-never-bare---dist-load&quot;&gt;Why &lt;code dir=&quot;auto&quot;&gt;--dist loadfile&lt;/code&gt; (never bare &lt;code dir=&quot;auto&quot;&gt;--dist load&lt;/code&gt;)&lt;/h2&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;pytest-xdist&lt;/code&gt; supports several distribution modes. We always use &lt;code dir=&quot;auto&quot;&gt;loadfile&lt;/code&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;loadfile&lt;/code&gt;&lt;/strong&gt; keeps every test that lives in the same file on a single
worker. File-scoped fixtures (&lt;code dir=&quot;auto&quot;&gt;scope=&quot;module&quot;&lt;/code&gt;, file-level collection
ordering, shared module state) keep working exactly as they do serially.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;load&lt;/code&gt;&lt;/strong&gt; (the bare default) scatters a single file’s tests across multiple
workers. That breaks file-scoped fixtures and any test that relies on
collection order within a file.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For that reason: &lt;strong&gt;always pass &lt;code dir=&quot;auto&quot;&gt;--dist loadfile&lt;/code&gt;; never use bare &lt;code dir=&quot;auto&quot;&gt;--dist load&lt;/code&gt;.&lt;/strong&gt; CI uses &lt;code dir=&quot;auto&quot;&gt;loadfile&lt;/code&gt; for the same reason.&lt;/p&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;-p no:cacheprovider&lt;/code&gt; disables pytest’s cache plugin so a parallel run never
races on the shared &lt;code dir=&quot;auto&quot;&gt;.pytest_cache&lt;/code&gt; directory.&lt;/p&gt;
&lt;h2 id=&quot;per-worker-home-isolation-the-master-enabler&quot;&gt;Per-worker HOME isolation (the master enabler)&lt;/h2&gt;
&lt;p&gt;A parallel run &lt;strong&gt;never touches the real &lt;code dir=&quot;auto&quot;&gt;~/.spec-kitty&lt;/code&gt;&lt;/strong&gt;. Each &lt;code dir=&quot;auto&quot;&gt;pytest-xdist&lt;/code&gt;
worker — and the serial “master” run when you omit &lt;code dir=&quot;auto&quot;&gt;-n auto&lt;/code&gt; — gets its own
isolated home directory. The isolation is set up in &lt;code dir=&quot;auto&quot;&gt;tests/conftest.py&lt;/code&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;pytest_configure&lt;/code&gt; points &lt;code dir=&quot;auto&quot;&gt;HOME&lt;/code&gt; / &lt;code dir=&quot;auto&quot;&gt;USERPROFILE&lt;/code&gt; and the XDG dirs
(&lt;code dir=&quot;auto&quot;&gt;XDG_CONFIG_HOME&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;XDG_DATA_HOME&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;XDG_STATE_HOME&lt;/code&gt;) at a per-worker base
&lt;strong&gt;before collection&lt;/strong&gt;, so modules that bind a home-derived path at import time
(for example paths derived from &lt;code dir=&quot;auto&quot;&gt;get_runtime_root()&lt;/code&gt;) resolve into the
isolated home.&lt;/li&gt;
&lt;li&gt;An autouse, function-scoped fixture re-asserts the &lt;code dir=&quot;auto&quot;&gt;HOME&lt;/code&gt; / &lt;code dir=&quot;auto&quot;&gt;USERPROFILE&lt;/code&gt; / XDG
env vars for every test, keyed by worker id, so call-time &lt;code dir=&quot;auto&quot;&gt;Path.home()&lt;/code&gt; reads
are isolated too. It does &lt;strong&gt;not&lt;/strong&gt; monkeypatch &lt;code dir=&quot;auto&quot;&gt;Path.home&lt;/code&gt; (that approach was the
cycle-1 regression that broke ~16 &lt;code dir=&quot;auto&quot;&gt;tests/sync&lt;/code&gt; cases — the fixture relies on
&lt;code dir=&quot;auto&quot;&gt;Path.home()&lt;/code&gt; natively resolving &lt;code dir=&quot;auto&quot;&gt;HOME&lt;/code&gt; via &lt;code dir=&quot;auto&quot;&gt;expanduser&lt;/code&gt;), so a test that sets
up its own tmp home via &lt;code dir=&quot;auto&quot;&gt;setenv(&apos;HOME&apos;, ...)&lt;/code&gt; cleanly overrides the per-worker
baseline.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The per-worker base is keyed by the xdist test-run UID and the worker id, so two
workers in the same run get distinct homes (no collision) and successive runs do
not reuse stale state. The regression guard
&lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_real_home_isolation_guard.py&lt;/code&gt; (SC-006) and
&lt;code dir=&quot;auto&quot;&gt;tests/test_worker_home_isolation.py&lt;/code&gt; prove this invariant.&lt;/p&gt;
&lt;p&gt;Because the real &lt;code dir=&quot;auto&quot;&gt;~/.spec-kitty&lt;/code&gt; is never bound, you do not need to back it up or
worry about a parallel run overwriting your real auth tokens or config.&lt;/p&gt;
&lt;h2 id=&quot;per-run-pytest-temp-root&quot;&gt;Per-run pytest temp root&lt;/h2&gt;
&lt;p&gt;Every pytest invocation also gets its &lt;strong&gt;own private basetemp&lt;/strong&gt; instead of the
shared &lt;code dir=&quot;auto&quot;&gt;/tmp/pytest-of-&amp;#x3C;user&gt;&lt;/code&gt; numbered tree: &lt;code dir=&quot;auto&quot;&gt;&amp;#x3C;temproot&gt;/spec-kitty-pytest-tmp/run-&amp;#x3C;pid&gt;&lt;/code&gt;
(&lt;code dir=&quot;auto&quot;&gt;temproot&lt;/code&gt; honors &lt;code dir=&quot;auto&quot;&gt;PYTEST_DEBUG_TEMPROOT&lt;/code&gt;, then &lt;code dir=&quot;auto&quot;&gt;TMPDIR&lt;/code&gt;). This is a
housekeeping fix, not a #63 fix: pytest’s default tree only ever shrinks
through its own locked, timeout-gated pruning, so a long-lived box accumulates
stale &lt;code dir=&quot;auto&quot;&gt;pytest-&amp;#x3C;N&gt;&lt;/code&gt; roots (some with live &lt;code dir=&quot;auto&quot;&gt;.lock&lt;/code&gt; files) that nobody notices
until someone clears them by hand. A controller-qa audit falsified the
original claim that this also fixed #63’s &lt;code dir=&quot;auto&quot;&gt;OSError: could not create numbered dir … after 10 tries&lt;/code&gt; crash — that crash names a &lt;em&gt;worker&lt;/em&gt; basetemp, which
takes a code path with no locking or pruning to race in the first place;
#63’s actual driver is &lt;code dir=&quot;auto&quot;&gt;tmp_path_retention_policy&lt;/code&gt; (see below), and it remains
open pending a full-suite run that reports a summary.&lt;/p&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;tests/conftest.py&lt;/code&gt; wires this in &lt;code dir=&quot;auto&quot;&gt;pytest_configure&lt;/code&gt; via
&lt;code dir=&quot;auto&quot;&gt;tests/_support/run_basetemp.py&lt;/code&gt;; xdist workers nest under the controller’s
choice as usual (&lt;code dir=&quot;auto&quot;&gt;…/run-&amp;#x3C;pid&gt;/popen-gwN&lt;/code&gt;). Consequences worth knowing:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;An explicit &lt;code dir=&quot;auto&quot;&gt;--basetemp&lt;/code&gt; still wins untouched — whoever passes one owns its
lifecycle.&lt;/li&gt;
&lt;li&gt;The basetemp (and the per-worker home base) is a &lt;strong&gt;git ceiling&lt;/strong&gt; (#7022):
&lt;code dir=&quot;auto&quot;&gt;pytest_configure&lt;/code&gt; appends both to &lt;code dir=&quot;auto&quot;&gt;GIT_CEILING_DIRECTORIES&lt;/code&gt;
(&lt;code dir=&quot;auto&quot;&gt;tests/_support/git_ceiling.py&lt;/code&gt;), so discovery from a &lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; never
climbs into a git work tree that happens to enclose the temp root (an agent
sandbox with &lt;code dir=&quot;auto&quot;&gt;TMPDIR&lt;/code&gt; under a checkout, a &lt;code dir=&quot;auto&quot;&gt;/tmp&lt;/code&gt; someone ran &lt;code dir=&quot;auto&quot;&gt;git init&lt;/code&gt; in).
Git and the migrated parent-enumerating repo/project-root walks in the stack
honor it, through the one walk in &lt;code dir=&quot;auto&quot;&gt;kernel.ancestor_walk&lt;/code&gt; (#7056); a repository
a test creates inside its own &lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; is still found. The &lt;code dir=&quot;auto&quot;&gt;while&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;range&lt;/code&gt;
loop walks (#7269) and &lt;code dir=&quot;auto&quot;&gt;session_start&lt;/code&gt;’s own walk with its &lt;code dir=&quot;auto&quot;&gt;stop=&lt;/code&gt; (#7122) are
not on it yet. It is the floor: a test
may still set its own, narrower ceiling with &lt;code dir=&quot;auto&quot;&gt;monkeypatch.setenv&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;tempfile&lt;/code&gt; lives inside the ceiling too&lt;/strong&gt; (#7076). The ceiling only governs
directories &lt;em&gt;below&lt;/em&gt; a listed entry, and &lt;code dir=&quot;auto&quot;&gt;tempfile&lt;/code&gt;’s root is the platform temp
root — above every entry — so without this every &lt;code dir=&quot;auto&quot;&gt;TemporaryDirectory&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;mkdtemp&lt;/code&gt;
in the suite (59 modules) made its directory outside the harness boundary, and
on a host whose &lt;code dir=&quot;auto&quot;&gt;/tmp&lt;/code&gt; had gained a stray &lt;code dir=&quot;auto&quot;&gt;.git&lt;/code&gt; those directories were inside a
foreign repository. &lt;code dir=&quot;auto&quot;&gt;pytest_configure&lt;/code&gt; now points &lt;code dir=&quot;auto&quot;&gt;TMPDIR&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;TEMP&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;TMP&lt;/code&gt; at
&lt;code dir=&quot;auto&quot;&gt;&amp;#x3C;basetemp&gt;/tmp&lt;/code&gt;, creates it, and resets &lt;code dir=&quot;auto&quot;&gt;tempfile.tempdir&lt;/code&gt;
(&lt;code dir=&quot;auto&quot;&gt;tests/_support/temp_root.py&lt;/code&gt;), so temporaries inherit the ceiling &lt;em&gt;and&lt;/em&gt; the
run’s own wipe/reap — there is no second cleanup path for them. pytest removes
and recreates a &lt;em&gt;given&lt;/em&gt; basetemp on first use, so a guard fixture re-creates the
root once that has happened; without it &lt;code dir=&quot;auto&quot;&gt;tempfile&lt;/code&gt; silently falls back to the
platform temp root.&lt;/li&gt;
&lt;li&gt;A test that builds a child environment &lt;strong&gt;from scratch&lt;/strong&gt; (an allowlist, or a copy
with &lt;code dir=&quot;auto&quot;&gt;GIT_*&lt;/code&gt; stripped) must carry the ceiling over itself: merge
&lt;code dir=&quot;auto&quot;&gt;tests/_support/temp_root.py::harness_env()&lt;/code&gt;, which yields the ceiling and the
temp-root variables and nothing else. Six sites do; they are pinned by
&lt;code dir=&quot;auto&quot;&gt;tests/unit/test_harness_child_env.py&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Retention is outcome-gated (#76): a healthy run (&lt;code dir=&quot;auto&quot;&gt;ExitCode.OK&lt;/code&gt;) leaves
nothing behind — it removes the run’s dir at interpreter exit. A run with
failures, errors, or an interruption keeps its private basetemp tree instead,
so &lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; contents from that run ARE available for post-mortem inspection,
bounded by the 24 h stale-crash sweep the next run performs at controller
startup (&lt;code dir=&quot;auto&quot;&gt;STALE_RUN_MAX_AGE_S&lt;/code&gt;, swept from &lt;code dir=&quot;auto&quot;&gt;install_run_basetemp&lt;/code&gt;; it also
catches a run that never reaches &lt;code dir=&quot;auto&quot;&gt;pytest_sessionfinish&lt;/code&gt;, e.g. a SIGKILL,
since the exit-gated reap never runs for it either). This differs from
pytest’s own default 3-session retention, which keeps the last three
sessions’ trees regardless of outcome.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;tmp_path-retention-policy&quot;&gt;&lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; retention policy&lt;/h2&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt; sets &lt;code dir=&quot;auto&quot;&gt;tmp_path_retention_policy = failed&lt;/code&gt;: a passing test’s
&lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; is removed in the &lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; fixture’s own teardown, right after
that test runs, instead of staying alive for the whole session (pytest’s
&lt;code dir=&quot;auto&quot;&gt;all&lt;/code&gt; default). Across the ~40k tests &lt;code dir=&quot;auto&quot;&gt;make test-full&lt;/code&gt; collects, retaining
every one’s &lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; for the full session is enough on its own to exhaust a
runner’s temp filesystem — the diagnosed driver behind #63’s pre-summary
crashes. Unlike the &lt;em&gt;session-end&lt;/em&gt; basetemp wipe (which pytest only does when
&lt;code dir=&quot;auto&quot;&gt;--basetemp&lt;/code&gt; was never given explicitly), this per-test teardown removal is
unconditional — it fires exactly the same whether the run’s basetemp came
from &lt;code dir=&quot;auto&quot;&gt;--basetemp&lt;/code&gt;, from this repo’s per-run private root above, or from
neither, which is why it still drains the pressure under this repo’s scheme
even though a basetemp is effectively always “given” here.&lt;/p&gt;
&lt;p&gt;The &lt;em&gt;other&lt;/em&gt; retention knob, &lt;code dir=&quot;auto&quot;&gt;tmp_path_retention_count&lt;/code&gt; (“how many past
sessions’ worth of numbered &lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; dirs to keep”), is a no-op in this
repo: it only ever applies to the code path that prunes pytest’s own
default numbered basetemp tree, which never runs once a basetemp is given —
so it is not a lever worth reaching for here.&lt;/p&gt;
&lt;h2 id=&quot;the-serial-marker-passes&quot;&gt;The serial marker passes&lt;/h2&gt;
&lt;p&gt;Per-worker HOME isolation protects per-user state, but it does &lt;strong&gt;not&lt;/strong&gt; protect
wall-clock timing or real multi-process/subprocess concurrency. Two marker
families are corrupted by co-scheduled &lt;code dir=&quot;auto&quot;&gt;pytest-xdist&lt;/code&gt; workers and get their
own dedicated serial pass instead of running in the main parallel pool:
&lt;code dir=&quot;auto&quot;&gt;stress&lt;/code&gt; (spawns real multi-process/subprocess concurrency) and &lt;code dir=&quot;auto&quot;&gt;timing&lt;/code&gt;
(measures wall-clock). &lt;code dir=&quot;auto&quot;&gt;make test-full&lt;/code&gt; deselects both from the parallel pass
(&lt;code dir=&quot;auto&quot;&gt;-m &quot;not stress and not timing&quot;&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt;’s parallel-unsafe marker set)
and then runs each family on its own:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;PWHEADLESS&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#984E4D&quot;&gt;stress and not windows_ci&lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n0&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;PWHEADLESS&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;timing&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n0&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;-n0&lt;/code&gt; forces serial execution even when xdist is installed. The per-test
timeout that guards a hung fork/process from stalling the pass indefinitely is
no longer passed on these commands: since #3143 it comes from the single
per-test timeout authority in &lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt; (&lt;code dir=&quot;auto&quot;&gt;timeout = 240&lt;/code&gt;), which applies to
every pass — parallel and serial alike. See
&lt;a href=&quot;#the-per-test-timeout&quot;&gt;The per-test timeout&lt;/a&gt; below. These mirror the
&lt;code dir=&quot;auto&quot;&gt;Makefile&lt;/code&gt;’s &lt;code dir=&quot;auto&quot;&gt;test-full&lt;/code&gt; target’s serial marker passes.&lt;/p&gt;
&lt;p&gt;(The former fourth pass ran the deleted sync daemon’s fixed-port suites — five
files bound to the reserved 9400–9449 port range, keyed off
&lt;code dir=&quot;auto&quot;&gt;tests/_real_port_suites.py&lt;/code&gt;’s &lt;code dir=&quot;auto&quot;&gt;FIXED_RANGE_SUITES&lt;/code&gt; registry, isolated so two
workers never contended for the same fixed port. Both the registry and the
sync daemon it protected died with the sync transport, issue #5; there is no
fixed-daemon-port family left to isolate.)&lt;/p&gt;
&lt;h2 id=&quot;the-per-test-timeout&quot;&gt;The per-test timeout&lt;/h2&gt;
&lt;p&gt;Every test has a &lt;strong&gt;240-second per-test timeout&lt;/strong&gt;, set once in &lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt;:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;ini&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt;[pytest]&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;timeout&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; = 240&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;This is the single per-test timeout authority (#3143). Because it lives in
&lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt;, it applies uniformly wherever pytest reads that file — the Linux
module shards, the Windows job (&lt;code dir=&quot;auto&quot;&gt;ci-windows.yml&lt;/code&gt;), a bare local &lt;code dir=&quot;auto&quot;&gt;pytest&lt;/code&gt;, and
the &lt;code dir=&quot;auto&quot;&gt;make test-fast&lt;/code&gt; / &lt;code dir=&quot;auto&quot;&gt;make test-full&lt;/code&gt; targets — so no surface is left without
one. Before #3143 the flags lived only in CI job definitions (and, briefly, only
in the Makefile serial passes), so Windows CI and every local run had no
per-test timeout at all.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;What a hang looks like now.&lt;/strong&gt; When a test runs longer than 240s, pytest-timeout
fails &lt;strong&gt;that one test by name&lt;/strong&gt; and the run continues to its summary:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#d6deeb;--1:#403f53&quot;&gt;+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#d6deeb;--1:#403f53&quot;&gt;...&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#d6deeb;--1:#403f53&quot;&gt;FAILED tests/some/module/test_thing.py::test_that_hung - Failed: Timeout &gt;240.0s&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;Contrast the old failure mode: a hang was killed by a job-level
&lt;code dir=&quot;auto&quot;&gt;timeout-minutes&lt;/code&gt;, ending the run with &lt;strong&gt;no summary, no counter, and no named
test&lt;/strong&gt; — you could not tell which test wedged.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Method is per platform, by design.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt; deliberately leaves
&lt;code dir=&quot;auto&quot;&gt;timeout_method&lt;/code&gt; unset. pytest-timeout then uses:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;signal&lt;/code&gt;&lt;/strong&gt; (SIGALRM) on POSIX — the Linux shards and local macOS/Linux runs.
This interrupts the running test at the timeout and prints the counted,
test-named &lt;code dir=&quot;auto&quot;&gt;Failed: Timeout &gt;240.0s&lt;/code&gt; line above.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;thread&lt;/code&gt;&lt;/strong&gt; on Windows, which has no SIGALRM. A hung &lt;code dir=&quot;auto&quot;&gt;windows_ci&lt;/code&gt; test still
fails by name with a thread-stack dump. The thread watchdog cannot interrupt a
wedged native (C) call, so a genuinely stuck native call may still fall through
to the job-level &lt;code dir=&quot;auto&quot;&gt;timeout-minutes: 20&lt;/code&gt;; re-run such a hang on Linux to get the
signal-method, counted failure. &lt;code dir=&quot;auto&quot;&gt;ci-windows.yml&lt;/code&gt; carries this note inline.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;240s is a runaway backstop, not a budget.&lt;/strong&gt; It sits far above the longest
observed test (~30s on recent shard timing). A legitimately long test should
carry its own &lt;code dir=&quot;auto&quot;&gt;@pytest.mark.timeout(&amp;#x3C;seconds&gt;)&lt;/code&gt; marker rather than raising this
floor; that marker (and any explicit command-line &lt;code dir=&quot;auto&quot;&gt;--timeout=&lt;/code&gt;, e.g. the
mutation-testing args) overrides the ini default. The default is pinned by
&lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_pytest_ini_timeout_default.py&lt;/code&gt; so it cannot silently
drift back out.&lt;/p&gt;
&lt;h2 id=&quot;volume-env-gates-spec_kitty_ulid_volume_full&quot;&gt;Volume env gates (&lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_ULID_VOLUME_FULL&lt;/code&gt;)&lt;/h2&gt;
&lt;p&gt;Some tests exercise large-volume ULID generation. By default they run at a
&lt;strong&gt;reduced&lt;/strong&gt; scale so the local default stays fast; the full scale is reachable
via an env gate (and is exercised on the nightly/full path). The assertion logic
is identical across scales — only the volume changes.&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;&amp;#x3C;ulid_test&gt;&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt;                               &lt;/span&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# reduced (fast, default)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;SPEC_KITTY_ULID_VOLUME_FULL&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;&amp;#x3C;ulid_test&gt;&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# full (nightly parity)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;h2 id=&quot;running-the-stability-ratchet-locally&quot;&gt;Running the stability ratchet locally&lt;/h2&gt;
&lt;p&gt;Before any shard is flipped to parallel, it must pass the stability ratchet
(C-RATCHET): N consecutive green parallel runs with no new flakes. The same
entrypoint CI uses is available locally (the WP02 coverage-safety harness):&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;python&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests._support.coverage_safety.ratchet&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#F78C6C;--1:#AA0982&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;--&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/agent&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#984E4D&quot;&gt;not slow&lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;Exit code &lt;code dir=&quot;auto&quot;&gt;0&lt;/code&gt; means all N runs were green and the flip is accepted; &lt;code dir=&quot;auto&quot;&gt;1&lt;/code&gt; means it
was rejected and the summary names any new or flaky failures. The Python API is
&lt;code dir=&quot;auto&quot;&gt;run_ratchet(...)&lt;/code&gt; from &lt;code dir=&quot;auto&quot;&gt;tests._support.coverage_safety&lt;/code&gt;. See
&lt;code dir=&quot;auto&quot;&gt;tests/_support/coverage_safety/README.md&lt;/code&gt; for the full harness (collection
equivalence and anti-vacuity mutation checks).&lt;/p&gt;
&lt;h2 id=&quot;validate-the-acceleration-copy-pasteable&quot;&gt;Validate the acceleration (copy-pasteable)&lt;/h2&gt;
&lt;p&gt;These are the mission’s reproducible validation steps. Run them from the repo
root to confirm the parallel run is coverage-neutral and at least 2× faster than
serial. (&lt;code dir=&quot;auto&quot;&gt;.venv/bin/pytest&lt;/code&gt; is the synced project interpreter; substitute
&lt;code dir=&quot;auto&quot;&gt;pytest&lt;/code&gt; if you run it directly.)&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 1. Serial baseline (whole-suite wall clock) and a per-shard nodeid reference.&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;time&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; .venv/bin/pytest tests/ -q -p no:cacheprovider     &lt;/span&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# serial baseline&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;.venv/bin/pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/charter&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;--collect-only&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#7FDBCA;--1:#096E72&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;sort&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#7FDBCA;--1:#096E72&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;/tmp/charter-serial.nodeids&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 2. Collection equivalence: serial vs parallel must collect identical nodeids.&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;.venv/bin/pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/charter&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;auto&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;--dist&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;loadfile&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;--collect-only&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#F78C6C;--1:#AA0982&quot;&gt;\&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span class=&quot;indent&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;--0:#7FDBCA;--1:#096E72&quot;&gt;|&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;sort&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#7FDBCA;--1:#096E72&quot;&gt;&gt;&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;/tmp/charter-par.nodeids&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;diff&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;/tmp/charter-serial.nodeids&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;/tmp/charter-par.nodeids&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt;   &lt;/span&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# must be empty&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 3. Stability ratchet: 3 consecutive green parallel runs (the same gate CI uses).&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;python&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests._support.coverage_safety.ratchet&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#F78C6C;--1:#AA0982&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;--&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#F78C6C;--1:#AA0982&quot;&gt;\&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span class=&quot;indent&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/charter&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#984E4D&quot;&gt;fast and not windows_ci&lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 4. Parallel-vs-serial timing: target ≥2× faster on a ≥4-core machine.&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;time&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;PWHEADLESS&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;.venv/bin/pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;auto&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;--dist&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;loadfile&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-p&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;no:cacheprovider&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#F78C6C;--1:#AA0982&quot;&gt;\&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span class=&quot;indent&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#984E4D&quot;&gt;not stress and not timing&lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;time&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;PWHEADLESS&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;.venv/bin/pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#984E4D&quot;&gt;stress and not windows_ci&lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n0&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;time&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;PWHEADLESS&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;.venv/bin/pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;tests/&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;timing&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n0&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-q&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 5. Real home untouched: mtime/inode unchanged (or path still absent) after the run.&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;ls&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-la&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;~/.spec-kitty&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#7FDBCA;--1:#096E72&quot;&gt;2&gt;&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;/dev/null&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;h2 id=&quot;ci-worker-policy-and-the-nightly-architectural-backstop&quot;&gt;CI worker policy and the nightly architectural backstop&lt;/h2&gt;
&lt;p&gt;This section is for maintainers and CI operators who read a battery job log or edit a
battery command. It records what mission &lt;code dir=&quot;auto&quot;&gt;ci-runtime-stabilisation&lt;/code&gt;
(&lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/5510&quot;&gt;#5510&lt;/a&gt;) changed; the local recipes
above are unchanged.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;CI passes a literal &lt;code dir=&quot;auto&quot;&gt;-n 4&lt;/code&gt;.&lt;/strong&gt; On the 4-vCPU hosted runner &lt;code dir=&quot;auto&quot;&gt;-n auto&lt;/code&gt; resolved to the
physical core count (2), so the battery ran on half of the available parallelism. Four
battery-class commands now pass &lt;code dir=&quot;auto&quot;&gt;-n 4 --dist loadfile&lt;/code&gt;: the always-on
&lt;code dir=&quot;auto&quot;&gt;architectural-fast&lt;/code&gt; job, each leg of &lt;code dir=&quot;auto&quot;&gt;architectural-heavy&lt;/code&gt;, the nightly
&lt;code dir=&quot;auto&quot;&gt;architectural-backstop&lt;/code&gt;, and the Packs corpus suite. The value equals
&lt;code dir=&quot;auto&quot;&gt;special_tiers.architectural.workers&lt;/code&gt; in &lt;code dir=&quot;auto&quot;&gt;.github/ci-module-registry.yml&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;-q&lt;/code&gt; is not passed to those commands.&lt;/strong&gt; With &lt;code dir=&quot;auto&quot;&gt;-q&lt;/code&gt;, xdist does not print its worker
banner. Without it, the job log carries the line &lt;code dir=&quot;auto&quot;&gt;created: 4/4 workers&lt;/code&gt;, which is the
evidence that four workers started. If you add &lt;code dir=&quot;auto&quot;&gt;-q&lt;/code&gt; back, that evidence disappears.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A static guard holds both rules.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;tests/ci/test_xdist_worker_policy.py&lt;/code&gt; fails on
&lt;code dir=&quot;auto&quot;&gt;-n auto&lt;/code&gt;, on a worker count that differs from the registry, and on &lt;code dir=&quot;auto&quot;&gt;-q&lt;/code&gt; in those
commands. It reads the workflow files, so it needs no CI run.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Local targets keep &lt;code dir=&quot;auto&quot;&gt;-n auto&lt;/code&gt;.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;make test-fast&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;make test-full&lt;/code&gt; run on machines
with an unknown core count, so the literal is a CI-runner fact and not a local
recommendation. The &lt;code dir=&quot;auto&quot;&gt;-n auto&lt;/code&gt; recipes in this page stay correct.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The test list is collected once per job, before pytest.&lt;/strong&gt; The heavy battery legs
and the &lt;code dir=&quot;auto&quot;&gt;ci&lt;/code&gt; module shard that holds the &lt;code dir=&quot;auto&quot;&gt;live_universe&lt;/code&gt; test run a pre-test step
that collects the full list of tests and stores it, so no test collects inside its
own setup while four workers compete for the CPUs. A first run still collects once
per consuming job; a re-run restores the stored list. The step, the cache key, the
conditions that bypass the store and the &lt;code dir=&quot;auto&quot;&gt;check&lt;/code&gt; that fails a silent fallback are in
&lt;a href=&quot;/development/reference/ci-gate-mechanics/#stored-test-universe-collection&quot;&gt;Stored test-universe collection&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The nightly backstop.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;ci-nightly.yml&lt;/code&gt; job &lt;code dir=&quot;auto&quot;&gt;architectural-backstop&lt;/code&gt; runs the full
battery base selection (&lt;code dir=&quot;auto&quot;&gt;tests/architectural&lt;/code&gt; with the base marker expression and the same
deselects) in one plain pytest invocation, with no partition plugin, on Python 3.12 with a
40-minute timeout. It exists so that a green nightly means the whole battery ran, which was
not true when the nightly reached the battery only through the &lt;code dir=&quot;auto&quot;&gt;fast or unit&lt;/code&gt; interpreter
shard. It is part of &lt;code dir=&quot;auto&quot;&gt;nightly-summary&lt;/code&gt;. A red run fails the nightly conclusion that the
release gate reads, and it escalates to a deduplicated, triaged P0 issue under the suite
key &lt;code dir=&quot;auto&quot;&gt;architectural&lt;/code&gt;. The decision is recorded in the amendment to
&lt;a href=&quot;/adr/3.x/2026-09-26-1-ci-coverage-honesty/&quot;&gt;ADR 2026-09-26-1&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;ci-shard-topology-mission-status&quot;&gt;CI shard-topology mission status&lt;/h2&gt;
&lt;p&gt;Point-in-time mission-status snapshots (named mission IDs, CI-confirmation state, and
per-job PENDING wall-clock records for the &lt;code dir=&quot;auto&quot;&gt;ci-test-topology-performance-01KXBJRT&lt;/code&gt;
shard-topology re-flip) have moved to
&lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/blob/main/docs/archive/plans/engineering-notes/testing-parallel-ci-topology-status.md&quot;&gt;&lt;code dir=&quot;auto&quot;&gt;testing-parallel-ci-topology-status.md&lt;/code&gt;&lt;/a&gt;
in engineering notes — this how-to page stays focused on the durable local workflow above.&lt;/p&gt;
&lt;h2 id=&quot;reproducing-3115-the-folded-uuid-render-width-defect&quot;&gt;Reproducing #3115 (the folded-uuid render-width defect)&lt;/h2&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Historical recipe.&lt;/strong&gt; The &lt;code dir=&quot;auto&quot;&gt;sync&lt;/code&gt; surface this section reproduces a defect on was retired
with the local sync transport (Convergence #3881); &lt;code dir=&quot;auto&quot;&gt;src/specify_cli/cli/commands/sync.py&lt;/code&gt;
no longer exists. The recipe is retained as the record of how #3115 was isolated — the
&lt;code dir=&quot;auto&quot;&gt;rich&lt;/code&gt; dumb-terminal 80-column fold behavior it demonstrates is still the general lesson.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;sync status&lt;/code&gt; / &lt;code dir=&quot;auto&quot;&gt;sync doctor&lt;/code&gt; render a &lt;code dir=&quot;auto&quot;&gt;Project&lt;/code&gt; column with &lt;code dir=&quot;auto&quot;&gt;overflow=&quot;fold&quot;&lt;/code&gt;
(&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/cli/commands/sync.py:1440&lt;/code&gt;, deliberate). At an 80-column
console width a 36-character project uuid folds across two lines and stops
being a contiguous substring of the captured output, so a plain &lt;code dir=&quot;auto&quot;&gt;uuid in out&lt;/code&gt;
assertion fails even though the journal is fully populated. The 80-column
width comes from &lt;code dir=&quot;auto&quot;&gt;rich.console.Console.size&lt;/code&gt;’s &lt;code dir=&quot;auto&quot;&gt;if self.is_dumb_terminal:&lt;/code&gt;
branch, which returns the hardcoded &lt;code dir=&quot;auto&quot;&gt;ConsoleDimensions(80, 25)&lt;/code&gt; &lt;strong&gt;above&lt;/strong&gt; the
&lt;code dir=&quot;auto&quot;&gt;COLUMNS&lt;/code&gt; read — so &lt;code dir=&quot;auto&quot;&gt;COLUMNS&lt;/code&gt;, which the affected tests’ isolation fixture
already sets, is never consulted. &lt;code dir=&quot;auto&quot;&gt;is_terminal&lt;/code&gt; is true whenever &lt;code dir=&quot;auto&quot;&gt;FORCE_COLOR&lt;/code&gt;
is set to a non-empty value, and &lt;code dir=&quot;auto&quot;&gt;is_dumb_terminal&lt;/code&gt; additionally requires
&lt;code dir=&quot;auto&quot;&gt;TERM&lt;/code&gt; to be &lt;code dir=&quot;auto&quot;&gt;dumb&lt;/code&gt; or &lt;code dir=&quot;auto&quot;&gt;unknown&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This is reproduced with two environment variables, one victim file, one
process — no &lt;code dir=&quot;auto&quot;&gt;pytest-xdist&lt;/code&gt;, because &lt;code dir=&quot;auto&quot;&gt;--dist loadfile&lt;/code&gt;’s worker assignment is
dynamic and work-stealing, so a reproducer that depends on a particular
assignment would not be reproducible by construction.&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;TERM&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;dumb&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;FORCE_COLOR&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;./scripts/repro_3115_render_width.sh&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;This runs &lt;code dir=&quot;auto&quot;&gt;tests/cli/commands/test_sync_status_per_project_3030.py&lt;/code&gt; (4
collected tests) alone, in one process, and reds with:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#d6deeb;--1:#403f53&quot;&gt;FAILED tests/cli/commands/test_sync_status_per_project_3030.py::test_status_names_every_project_with_count_age_and_consent&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#d6deeb;--1:#403f53&quot;&gt;AssertionError: aaaaaaaa-0000-0000-0000-000000000001 is in the journal but `status` did not name it&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#d6deeb;--1:#403f53&quot;&gt;1 failed, 3 passed in ~46-56s&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;The sibling victim file reproduces the same defect independently:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;TERM&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;dumb&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;FORCE_COLOR&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;./scripts/repro_3115_render_width.sh&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;doctor&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;tests/cli/commands/test_sync_doctor_per_project_3030.py&lt;/code&gt; collects &lt;strong&gt;12&lt;/strong&gt;
tests and reds &lt;code dir=&quot;auto&quot;&gt;1 failed, 11 passed&lt;/code&gt;, with assertion text
&lt;code dir=&quot;auto&quot;&gt;aaaaaaaa-0000-0000-0000-000000000001 is in the journal but doctor did not name it&lt;/code&gt; (no backticks around &lt;code dir=&quot;auto&quot;&gt;doctor&lt;/code&gt; — the two victim files’ f-strings are
not identical, so quote each one’s text verbatim rather than assuming they
match).&lt;/p&gt;
&lt;p&gt;Only the first of the three seeded projects (&lt;code dir=&quot;auto&quot;&gt;CONSENTED&lt;/code&gt;,
&lt;code dir=&quot;auto&quot;&gt;aaaaaaaa-0000-0000-0000-000000000001&lt;/code&gt;) demonstrates the defect: the other two
(&lt;code dir=&quot;auto&quot;&gt;SILENT&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;OPTED_OUT&lt;/code&gt;) pass at width 80 anyway, via an un-tabled warning
paragraph that reprints their identity outside the folding table.
&lt;code dir=&quot;auto&quot;&gt;Queue 0 event(s)&lt;/code&gt; / &lt;code dir=&quot;auto&quot;&gt;Queue size 0 / 100,000&lt;/code&gt; appear in the captured output
regardless of outcome (&lt;code dir=&quot;auto&quot;&gt;OfflineQueue().size()&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;sync.py:5182-5185&lt;/code&gt;) — they
are &lt;strong&gt;not&lt;/strong&gt; a signature of this defect and must not be read as one.&lt;/p&gt;
&lt;p&gt;Both counted lines are collected-count-relative: &lt;code dir=&quot;auto&quot;&gt;test_sync_status_per_project_3030.py&lt;/code&gt;
collects 4 (red is &lt;code dir=&quot;auto&quot;&gt;1 failed, 3 passed&lt;/code&gt;), &lt;code dir=&quot;auto&quot;&gt;test_sync_doctor_per_project_3030.py&lt;/code&gt;
collects 12 (red is &lt;code dir=&quot;auto&quot;&gt;1 failed, 11 passed&lt;/code&gt;). A count line that does not
reconcile against its file’s own &lt;code dir=&quot;auto&quot;&gt;--collect-only -q&lt;/code&gt; count is not evidence.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Control&lt;/strong&gt; — the same command plus &lt;code dir=&quot;auto&quot;&gt;TTY_COMPATIBLE=0&lt;/code&gt; must PASS. This is what
distinguishes “the width is the cause” from “this file is just broken”:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;TERM&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;dumb&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;FORCE_COLOR&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;TTY_COMPATIBLE&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;./scripts/repro_3115_render_width.sh&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 4 passed&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;TERM&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;dumb&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;FORCE_COLOR&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;TTY_COMPATIBLE&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;./scripts/repro_3115_render_width.sh&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;doctor&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 12 passed&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;Determinism&lt;/strong&gt; is not established by repetition alone: &lt;code dir=&quot;auto&quot;&gt;pytest-randomly&lt;/code&gt; is
not installed on this tree, so nothing randomises order and “the same node-id
comes up red three times in a row” is trivially true regardless of whether the
red is order-dependent. The clause that can actually fail is running the
failing case &lt;strong&gt;alone, by node-id&lt;/strong&gt;, with no file-siblings collected first:&lt;/p&gt;
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;TERM&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;dumb&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;FORCE_COLOR&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;python3&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-m&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#F78C6C;--1:#AA0982&quot;&gt;\&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span class=&quot;indent&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#984E4D&quot;&gt;tests/cli/commands/test_sync_status_per_project_3030.py::test_status_names_every_project_with_count_age_and_consent&lt;/span&gt;&lt;span style=&quot;--0:#D9F5DD;--1:#111111&quot;&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#919F9F;--1:#5F636F&quot;&gt;# 1 failed, collected count 1, identical assertion text&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;A red that needs its file-siblings to run first would be order-dependent and
would not be reproducible by construction (C-004).&lt;/p&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;./scripts/repro_3115_render_width.sh&lt;/code&gt; computes its own repo root from its own
location and sets &lt;code dir=&quot;auto&quot;&gt;PYTHONPATH&lt;/code&gt; to that repo’s &lt;code dir=&quot;auto&quot;&gt;src/&lt;/code&gt; before invoking pytest.
This matters in a &lt;code dir=&quot;auto&quot;&gt;git worktree&lt;/code&gt;: the shared &lt;code dir=&quot;auto&quot;&gt;.venv&lt;/code&gt;’s
&lt;code dir=&quot;auto&quot;&gt;_editable_impl_spec_kitty_cli.pth&lt;/code&gt; holds the &lt;strong&gt;main checkout’s&lt;/strong&gt; absolute
&lt;code dir=&quot;auto&quot;&gt;src&lt;/code&gt; path, so a bare &lt;code dir=&quot;auto&quot;&gt;pytest&lt;/code&gt; run inside a worktree using that &lt;code dir=&quot;auto&quot;&gt;.venv&lt;/code&gt;
silently imports the main checkout’s live tree instead of the worktree’s own
source. The script’s &lt;code dir=&quot;auto&quot;&gt;PYTHONPATH&lt;/code&gt; line exists specifically to defeat that.&lt;/p&gt;
&lt;p&gt;The whole reproducer (one victim file, one process) completes in well under 2
minutes (measured around 46-59s per file on this codebase’s base commit,
&lt;code dir=&quot;auto&quot;&gt;bb2020fea9&lt;/code&gt;) — cheap enough that nothing downstream needs to take the defect
on faith.&lt;/p&gt;
&lt;h2 id=&quot;charter-write-commands-and-the-working-directory&quot;&gt;Charter write commands and the working directory&lt;/h2&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;generate&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;synthesize&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;resynthesize&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;activate&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;deactivate&lt;/code&gt; ask the
charter write guard whether the working directory is a linked worktree. A test
that aims one of them at a temporary project without moving the process working
directory there passes from a repository root checkout and fails from a linked
worktree. Move the working directory into the temporary project; the
&lt;code dir=&quot;auto&quot;&gt;charter_cwd_isolation&lt;/code&gt; fixture does this. A suite-wide check reports a test
that reaches the write guard at a path inside the invoking checkout as an error
at teardown, in every checkout, not only in a worktree.&lt;/p&gt;</content>
</entry>
