<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xml:base="https://docs.spec-kitty.ai/development/testing/testing-flakiness/">
  <id>https://docs.spec-kitty.ai/development/testing/testing-flakiness/</id>
  <title>Test-flakiness handling policy</title>
  <updated>2026-11-08T00:00:00.000Z</updated>
  <author><name>Spec Kitty</name></author>
  <link rel="alternate" type="text/html" href="https://docs.spec-kitty.ai/development/testing/testing-flakiness/"/>
  <summary>Spec Kitty&apos;s suite-wide flaky-test policy: how to detect a flake, what is and isn&apos;t allowed (never retry-to-green), and the disposition of each known flake surface.</summary>
  <content type="html">&lt;span id=&quot;test-flakiness-handling-policy&quot;&gt;&lt;/span&gt;
&lt;p&gt;A &lt;em&gt;flaky&lt;/em&gt; test is one whose pass/fail outcome changes between runs &lt;strong&gt;without any
change to the code under test&lt;/strong&gt; — it sometimes goes red on CI for reasons
unrelated to the diff under review. Flakes waste review cycles and, left
unmanaged, normalise a red CI that everyone learns to ignore.&lt;/p&gt;
&lt;p&gt;This page is the suite-wide policy for handling them: how we &lt;strong&gt;detect&lt;/strong&gt; a flake,
what we are allowed to do about it (and explicitly &lt;em&gt;not&lt;/em&gt; allowed to do), and the
current disposition of every known flake surface in this repo.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The one rule that governs everything below:&lt;/strong&gt; we never make a test pass by
&lt;em&gt;retrying it until it goes green&lt;/em&gt;. A green-after-retry is the
“fixed because it looks fixed” trap — it hides genuine regressions and is
incompatible with this repo’s live-evidence / anti-laziness discipline. Retry
plugins (&lt;code dir=&quot;auto&quot;&gt;pytest-rerunfailures&lt;/code&gt;, the PyPI &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; plugin) are therefore &lt;strong&gt;not
adopted&lt;/strong&gt;, and are not present in the dependency set.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2 id=&quot;️-naming-flaky-the-marker-is-not-flaky-the-concept&quot;&gt;⚠️ Naming: &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; the marker is &lt;em&gt;not&lt;/em&gt; “flaky” the concept&lt;/h2&gt;
&lt;p&gt;This repo already registers a &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; pytest marker, and it means something
&lt;strong&gt;narrow and unrelated&lt;/strong&gt; to this policy. Per
&lt;a href=&quot;/adr/3.x/2026-04-20-1-mutation-testing-as-local-only-quality-gate/&quot;&gt;ADR 2026-04-20-1&lt;/a&gt;,
&lt;code dir=&quot;auto&quot;&gt;@pytest.mark.flaky&lt;/code&gt; means &lt;em&gt;“passes reliably in the main suite but is
non-deterministic &lt;strong&gt;under &lt;code dir=&quot;auto&quot;&gt;mutmut&lt;/code&gt; / forking pipelines&lt;/strong&gt;”&lt;/em&gt; — it is a deselection
bucket for the mutation-testing sandbox, &lt;strong&gt;not&lt;/strong&gt; a CI-quarantine mechanism.&lt;/p&gt;
&lt;p&gt;Do not reach for the &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; marker to deal with a CI flake. If a genuine
quarantine mechanism is ever needed, it must use a &lt;strong&gt;different&lt;/strong&gt; name
(&lt;code dir=&quot;auto&quot;&gt;quarantine&lt;/code&gt;, see below) so the two never get conflated.&lt;/p&gt;
&lt;h2 id=&quot;the-three-tiers&quot;&gt;The three tiers&lt;/h2&gt;
&lt;p&gt;Every flake in this suite falls into one of three tiers, and each tier has a
single sanctioned response.&lt;/p&gt;





























&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Tier&lt;/th&gt;&lt;th&gt;What it is&lt;/th&gt;&lt;th&gt;Example&lt;/th&gt;&lt;th&gt;Sanctioned response&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;1. Threshold / budget gate&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;A test that asserts a measurement stays under a wall-clock / size budget. CI runners are shared and noisy, so a generous gate occasionally trips with no real regression.&lt;/td&gt;&lt;td&gt;Timing-budget tests (&lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_spec_kitty_home_pin_budget.py&lt;/code&gt;), the &lt;code dir=&quot;auto&quot;&gt;-m timing&lt;/code&gt; gate.&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Tune the budget — never retry.&lt;/strong&gt; Tune a budget only with evidence: a real regression adds time &lt;strong&gt;consistently&lt;/strong&gt; and still trips a calibrated gate; a retry would mask exactly that. A test that pays interpreter cold start (a fresh &lt;code dir=&quot;auto&quot;&gt;python -m specify_cli&lt;/code&gt; per run) does not take an absolute number of seconds at all: it asserts a &lt;strong&gt;runner-relative measure&lt;/strong&gt;, the ratio of its median to a start-up floor median sampled interleaved in the same run, with the limit in &lt;code dir=&quot;auto&quot;&gt;tests/_perf_helpers.py&lt;/code&gt; and calibrated from recorded data (see the audit row and &lt;a href=&quot;/adr/4.x/2026-10-05-2-runner-relative-performance-budgets/&quot;&gt;the decision record&lt;/a&gt;). Widening an absolute budget to absorb a slow runner is not accepted for these tests.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;2. Correctness test&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;A test of logical behaviour that should be 100% deterministic. If it flakes, the test (or the code) has a hidden nondeterminism — shared global state, ordering assumptions, fixture-teardown races, monkeypatch leakage, import-time side effects.&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/specify_cli/shims/test_registry.py&lt;/code&gt; (parallel-collection nondeterminism).&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Fix the root cause — never retry.&lt;/strong&gt; A correctness test that needs a retry is lying. Find the nondeterminism and remove it.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;strong&gt;3. Genuinely environmental&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;A test that depends on an OS-global resource — real TCP ports, singleton daemons, the real filesystem — that cannot be fully isolated per worker.&lt;/td&gt;&lt;td&gt;Historical real-port / daemon suites, deleted with the sync transport (issue #5).&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Surgical handling only.&lt;/strong&gt; First serialise (&lt;code dir=&quot;auto&quot;&gt;-n0&lt;/code&gt;) and isolate (per-worker HOME) — see &lt;a href=&quot;/development/testing/testing-parallel/&quot;&gt;testing-parallel.md&lt;/a&gt;. Only if a residual, irreducible environmental flake remains do we &lt;strong&gt;quarantine&lt;/strong&gt; (below) — never a blanket retry.&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;h2 id=&quot;detection-confirm-a-flake-before-you-treat-it&quot;&gt;Detection: confirm a flake before you treat it&lt;/h2&gt;
&lt;p&gt;One red run is a single data point — pytest alone cannot tell &lt;em&gt;flaky&lt;/em&gt; from
&lt;em&gt;broken&lt;/em&gt;. Before declaring a test “flaky” and applying any of the responses
above, &lt;strong&gt;reproduce the nondeterminism&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Re-run the test in isolation, and under the parallel runner it failed on:
&lt;div class=&quot;expressive-code&quot;&gt;&lt;figure class=&quot;frame is-terminal not-content&quot;&gt;&lt;figcaption class=&quot;header&quot;&gt;&lt;span class=&quot;title&quot;&gt;&lt;/span&gt;&lt;span class=&quot;sr-only&quot;&gt;Terminal window&lt;/span&gt;&lt;/figcaption&gt;&lt;pre data-language=&quot;bash&quot;&gt;&lt;code&gt;&lt;div class=&quot;ec-line&quot;&gt;&lt;div class=&quot;code&quot;&gt;&lt;span style=&quot;--0:#C5E478;--1:#3B61B0&quot;&gt;PWHEADLESS&lt;/span&gt;&lt;span style=&quot;--0:#C792EA;--1:#8844AE&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;pytest&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;&amp;#x3C;path&gt;&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-n&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;auto&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;--dist&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;loadfile&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#82AAFF;--1:#3B61B0&quot;&gt;-p&lt;/span&gt;&lt;span style=&quot;--0:#D6DEEB;--1:#403F53&quot;&gt; &lt;/span&gt;&lt;span style=&quot;--0:#ECC48D;--1:#3B61B0&quot;&gt;no:cacheprovider&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;/li&gt;
&lt;li&gt;For ordering / hash-seed flakes, re-run under different &lt;code dir=&quot;auto&quot;&gt;PYTHONHASHSEED&lt;/code&gt;
values and diff the collected node IDs.&lt;/li&gt;
&lt;li&gt;For a confidence signal, use the existing &lt;strong&gt;stability ratchet&lt;/strong&gt; — N consecutive
green parallel runs, the same gate CI uses for shard flips. The authoritative
invocation lives in
&lt;a href=&quot;/development/testing/testing-parallel/#running-the-stability-ratchet-locally&quot;&gt;testing-parallel.md → Running the stability ratchet locally&lt;/a&gt;;
point it at the suspect &lt;code dir=&quot;auto&quot;&gt;&amp;#x3C;path&gt;&lt;/code&gt; instead of fabricating a new command.
(Full harness: &lt;code dir=&quot;auto&quot;&gt;tests/_support/coverage_safety/README.md&lt;/code&gt;.)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A test that cannot be made to fail again under these probes is &lt;strong&gt;not&lt;/strong&gt; confirmed
flaky — do not annotate it.&lt;/p&gt;
&lt;h2 id=&quot;tooling-decision&quot;&gt;Tooling decision&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;No retry tooling — ever.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;pytest-rerunfailures&lt;/code&gt; / PyPI &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; are
deliberately not in the dependency set (see the rule at the top). Detection and
isolation reuse what the repo already has; the only thing built specifically for
this policy is the &lt;code dir=&quot;auto&quot;&gt;quarantine&lt;/code&gt; marker.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No retry plugin.&lt;/strong&gt; See the rule at the top.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Detection&lt;/strong&gt; uses the existing stability ratchet
(&lt;code dir=&quot;auto&quot;&gt;tests._support.coverage_safety.ratchet&lt;/code&gt;), not a new confidence tool.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Isolation&lt;/strong&gt; (the first line of defence for Tier 3) is the existing
per-worker HOME isolation + serial &lt;code dir=&quot;auto&quot;&gt;-n0&lt;/code&gt; pass documented in
&lt;a href=&quot;/development/testing/testing-parallel/&quot;&gt;testing-parallel.md&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;quarantine--built-env-gated-non-blocking&quot;&gt;&lt;code dir=&quot;auto&quot;&gt;quarantine&lt;/code&gt; — built, env-gated, non-blocking&lt;/h3&gt;
&lt;p&gt;The sanctioned mechanism for an irreducible Tier-3 flake is a dedicated
&lt;code dir=&quot;auto&quot;&gt;quarantine&lt;/code&gt; marker — &lt;strong&gt;not&lt;/strong&gt; a retry, and &lt;strong&gt;not&lt;/strong&gt; the existing &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; marker.
It is implemented as a single, un-bypassable chokepoint:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Registered&lt;/strong&gt; canonically in &lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt;’s &lt;code dir=&quot;auto&quot;&gt;markers&lt;/code&gt; block — the single
source of truth for the marker registry (#2034) — sufficient for
&lt;code dir=&quot;auto&quot;&gt;--strict-markers&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Held out of every normal run.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;tests/conftest.py&lt;/code&gt;’s
&lt;code dir=&quot;auto&quot;&gt;pytest_collection_modifyitems&lt;/code&gt; skips any &lt;code dir=&quot;auto&quot;&gt;@pytest.mark.quarantine&lt;/code&gt; test
&lt;strong&gt;unless &lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_RUN_QUARANTINE=1&lt;/code&gt;&lt;/strong&gt;. Because this is collection-time and
global, no &lt;code dir=&quot;auto&quot;&gt;-m&lt;/code&gt; selector in any CI job (present or future) can accidentally
run a quarantined test — the gate cannot be forgotten. The opt-in is strict
(only the literal &lt;code dir=&quot;auto&quot;&gt;&quot;1&quot;&lt;/code&gt;); the pure decision lives in
&lt;code dir=&quot;auto&quot;&gt;tests/_support/quarantine.py&lt;/code&gt; and is unit-tested.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Visible only by explicit opt-in; not currently scheduled in CI.&lt;/strong&gt; Setting
&lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_RUN_QUARANTINE=1&lt;/code&gt; and selecting &lt;code dir=&quot;auto&quot;&gt;-m quarantine&lt;/code&gt; runs a
quarantined test for real locally. The current hosted CI topology has one
&lt;code dir=&quot;auto&quot;&gt;suite&lt;/code&gt; job and no &lt;code dir=&quot;auto&quot;&gt;quarantine-visibility&lt;/code&gt; lane, so quarantine visibility is
&lt;strong&gt;not enforced there&lt;/strong&gt;. A PR that adds the first quarantined test must also
add a non-blocking visibility lane (or explicitly change this policy); it
must not rely on the normal suite, which skips quarantined tests.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;strong&gt;To quarantine a test:&lt;/strong&gt; mark it &lt;code dir=&quot;auto&quot;&gt;@pytest.mark.quarantine&lt;/code&gt; with a one-line
reason &lt;strong&gt;and a tracking-issue link&lt;/strong&gt; — every quarantined test is tech debt with
an owner. The wiring above (&lt;code dir=&quot;auto&quot;&gt;test_quarantine_marker.py&lt;/code&gt;) is enforced.&lt;/p&gt;
&lt;p&gt;As of this writing &lt;strong&gt;no test is quarantined&lt;/strong&gt; (see the disposition table — every
known surface is fixed, correctly handled, or retired). The mechanism exists so the first
irreducible flake has a sanctioned home instead of a retry.&lt;/p&gt;
&lt;h2 id=&quot;audit--disposition-of-known-flake-surfaces&quot;&gt;Audit + disposition of known flake surfaces&lt;/h2&gt;






































































&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Surface&lt;/th&gt;&lt;th&gt;Tier&lt;/th&gt;&lt;th&gt;Disposition&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/charter/activation/test_stored_spelling_6527.py::test_a_writer_that_names_only_in_force_artifacts_stores_nothing[cascade]&lt;/code&gt; (before #6685: &lt;code dir=&quot;auto&quot;&gt;test_writer_stores_stems_only[cascade]&lt;/code&gt;; over the 240 s per-test backstop)&lt;/td&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Root cause partly fixed; the interim budget stays until #6854&lt;/strong&gt; (#6777). The case is not a budget test: it drives a real &lt;code dir=&quot;auto&quot;&gt;charter activate --cascade all&lt;/code&gt;, and its cost is avoidable work. Part one is fixed: the cascade resolved each of its 114 targets with its own fresh identity pass, re-walking and re-parsing the whole artifact catalog per target (121 lookups, 5309 id reads, 5028 artifact-file parses); one shared &lt;code dir=&quot;auto&quot;&gt;ResolutionPass&lt;/code&gt; covering the whole batch before the first write takes it to 240 parses and 0.62–0.70× the wall clock (interleaved A/B; absolute seconds on a shared host are not evidence). Part two remains: each target’s &lt;code dir=&quot;auto&quot;&gt;ActiveCharterManager.activate()&lt;/code&gt; still rescans its kind’s catalog (114 scans where 6 would do), and with only part one fixed the case still crossed 240 s on a per-PR runner (run 42462186463). So #6776’s 600 s per-test budget (&lt;code dir=&quot;auto&quot;&gt;_CASCADE_TIMEOUT_S&lt;/code&gt;) is kept as a budget-gate tune, not a retry, beside the documented &lt;code dir=&quot;auto&quot;&gt;slow&lt;/code&gt; tier the module shards still select. Two guards pin it: one fails if any other case overrides the backstop or if the budget’s comment stops naming #6854 (so dropping it is a deliberate act when #6854 lands), and one fails if the batch ever stops sharing one pass or resolves after the first write. No retry was added.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Historical &lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_wp_prompt_build_latency.py&lt;/code&gt; NFR-002 latency surface&lt;/td&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Retired — this surface no longer exists.&lt;/strong&gt; The low-signal suite cleanup in PR #3285 removed the file. The current timing policy is carried by the live &lt;code dir=&quot;auto&quot;&gt;-m timing&lt;/code&gt; lane and its current budget tests; a future latency flake gets a new, separately evidenced row.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;doctor restart-daemon&lt;/code&gt; issue #1153 / NFR-002 wall-clock gate&lt;/td&gt;&lt;td&gt;3&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Retired as a portable CI performance claim in PR #3285.&lt;/strong&gt; The original ≤10-second end-to-end requirement depended on OS-global daemon and port state and repeatedly failed during hosted-macOS control-plane bootstrap without a corresponding product regression. The controlled Linux lane now enforces the replacement functional contract: the old daemon stops, a distinct PID starts, and the control plane becomes healthy within bounded subprocess and readiness timeouts. macOS remains supported through platform-independent tests and local smoke evidence; a future cross-platform restart performance SLO requires a controlled developer/canary harness because shared-runner wall clock is not accepted as evidence.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/sync/test_orphan_sweep.py&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;tests/_real_port_suites.py&lt;/code&gt;’s &lt;code dir=&quot;auto&quot;&gt;FIXED_RANGE_SUITES&lt;/code&gt; (real fixed ports 9400–9449, daemons)&lt;/td&gt;&lt;td&gt;3&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Retired — this surface no longer exists.&lt;/strong&gt; The sync transport that owned it was deleted (issue #5): &lt;code dir=&quot;auto&quot;&gt;tests/sync/&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;tests/_real_port_suites.py&lt;/code&gt; are gone from the repo, so there is nothing left to serialise or quarantine here. &lt;code dir=&quot;auto&quot;&gt;make test-full&lt;/code&gt;’s current topology (one parallel pass plus dedicated &lt;code dir=&quot;auto&quot;&gt;-n0&lt;/code&gt; serial passes for &lt;code dir=&quot;auto&quot;&gt;stress&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;timing&lt;/code&gt;) is described in &lt;code dir=&quot;auto&quot;&gt;CLAUDE.md&lt;/code&gt;’s Commands section.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/performance/test_owned_checkout_perf.py&lt;/code&gt; (three owned-checkout commands), &lt;code dir=&quot;auto&quot;&gt;test_cli_startup_budget_4409.py&lt;/code&gt; (&lt;code dir=&quot;auto&quot;&gt;--help&lt;/code&gt; start-up) and &lt;code dir=&quot;auto&quot;&gt;test_cli_startup_agent_commands_freshness.py&lt;/code&gt; (warm repeated leaf command), nightly &lt;code dir=&quot;auto&quot;&gt;performance&lt;/code&gt; job, #5419 / #5614&lt;/td&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Fixed by a runner-relative measure.&lt;/strong&gt; The three owned-checkout tests compared a median with an absolute 2.5 s and went red on four of six nightly runs with no product change: the same bare &lt;code dir=&quot;auto&quot;&gt;--help&lt;/code&gt; moved from 1.14 s to 1.95 s between runs. They now assert the ratio of the command median to a start-up floor median (&lt;code dir=&quot;auto&quot;&gt;--version&lt;/code&gt;), five spawns of each interleaved in the same run, against &lt;code dir=&quot;auto&quot;&gt;OWNED_CHECKOUT_RATIO_LIMIT&lt;/code&gt;; the two start-up tests assert the ratio of the start-up median to a fixed interpreter workload measured in the same run (&lt;code dir=&quot;auto&quot;&gt;STARTUP_RATIO_LIMIT&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;WARM_LEAF_RATIO_LIMIT&lt;/code&gt;). No absolute number of seconds remains. Every limit lives in &lt;code dir=&quot;auto&quot;&gt;tests/_perf_helpers.py&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_perf_limit_authority.py&lt;/code&gt; fails when another test file defines one. The limits come from recorded clean and planted ratios (idle and CPU-throttled), with their headroom, in the &lt;a href=&quot;/adr/4.x/2026-10-05-2-runner-relative-performance-budgets/&quot;&gt;decision record&lt;/a&gt;; committed planted-work tests prove each relative assertion can fail. No retry was added.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_same_tier_uniqueness.py&lt;/code&gt; universe fixture (16 &lt;code dir=&quot;auto&quot;&gt;Failed: Timeout (&gt;240.0s)&lt;/code&gt; errors in the architectural battery 2/2 leg, #7052)&lt;/td&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Fixed by tuning the budget, with evidence.&lt;/strong&gt; The module-scoped &lt;code dir=&quot;auto&quot;&gt;universe&lt;/code&gt; fixture reuses the stored test-universe record (0.6 s) but collects the whole tree itself when the store is bypassed (a dirty checkout, no record): 89-233 s for a first CI run, 247.7 s on the reporting machine, 90-422 s on a shared development VM. That real collection sat at or over the suite’s 240 s per-test backstop, so it was killed and every test depending on the fixture errored at once. &lt;code dir=&quot;auto&quot;&gt;_gate_coverage.COLLECT_TIMEOUT_S&lt;/code&gt; now names the collection ceiling once and the tests whose fixture closure reaches the universe carry &lt;code dir=&quot;auto&quot;&gt;@pytest.mark.timeout(COLLECT_TIMEOUT_S + 300)&lt;/code&gt;; the suite-wide 240 s default is unchanged, a test that never collects keeps it, and a guard test fails if a consumer’s budget is missing or below the ceiling or if a collection subprocess repeats the number as a literal. No retry was added, and a bypass on CI is still a &lt;code dir=&quot;auto&quot;&gt;check&lt;/code&gt;-step failure with its reason and dirty paths. See &lt;a href=&quot;/development/reference/ci-gate-mechanics/&quot;&gt;ci-gate-mechanics&lt;/a&gt;.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Nightly &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd&lt;/code&gt; scenario “Consolidating lands the approved work and leaves my untracked files alone” (&lt;code dir=&quot;auto&quot;&gt;tests/scenarios/consolidate/consolidate.feature&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;CliTimeoutException&lt;/code&gt; after the driver’s 2-minute &lt;code dir=&quot;auto&quot;&gt;DEFAULT_TIMEOUT&lt;/code&gt; on &lt;code dir=&quot;auto&quot;&gt;windows-latest&lt;/code&gt;, run 44009807449), #7689&lt;/td&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Root cause partly fixed, then the budget tuned with evidence.&lt;/strong&gt; Not a hang: the &lt;code dir=&quot;auto&quot;&gt;consolidate&lt;/code&gt; process itself was still running at 120 s (the &lt;code dir=&quot;auto&quot;&gt;waitFor&lt;/code&gt; branch, not the pipe drain), no stdin prompt is reachable with &lt;code dir=&quot;auto&quot;&gt;--yes&lt;/code&gt;, and no commit in the green-to-red window changed its work (671 &lt;code dir=&quot;auto&quot;&gt;git&lt;/code&gt; processes at both SHAs, 15-21 s on Linux). The red run’s own &lt;code dir=&quot;auto&quot;&gt;cucumber.json&lt;/code&gt; shows that Windows runner 7-9x slower than Linux on every CLI step, which puts &lt;code dir=&quot;auto&quot;&gt;consolidate&lt;/code&gt; at about 105-165 s. Part one is fixed: a run-scoped git read cache (&lt;code dir=&quot;auto&quot;&gt;kernel.git.read_cache&lt;/code&gt;, entered only by &lt;code dir=&quot;auto&quot;&gt;consolidate&lt;/code&gt;) answers repeated reads without a process while the files they depend on are unchanged, taking the same run from 671 to 470 &lt;code dir=&quot;auto&quot;&gt;git&lt;/code&gt; processes; &lt;code dir=&quot;auto&quot;&gt;tests/integration/test_7689_consolidate_git_spawn_ceiling.py&lt;/code&gt; keeps it at or below 500. That alone gives no safe margin on a slow Windows runner, so &lt;code dir=&quot;auto&quot;&gt;CliProcessDriver.CONSOLIDATE_TIMEOUT&lt;/code&gt; (4 minutes, about 1.5x the worst estimate) replaces the shared 2-minute ceiling for &lt;code dir=&quot;auto&quot;&gt;spec-kitty consolidate&lt;/code&gt; only; every other command keeps &lt;code dir=&quot;auto&quot;&gt;DEFAULT_TIMEOUT&lt;/code&gt;. &lt;code dir=&quot;auto&quot;&gt;tests/ci/test_acceptance_cli_ceilings.py&lt;/code&gt; fails if the default moves, if a named ceiling is not above it or exceeds &lt;code dir=&quot;auto&quot;&gt;MAX_COMMAND_TIMEOUT&lt;/code&gt; (at most 5 minutes), if another command gains a named ceiling, or if a ceiling’s comment or this row stops naming its issue. No retry was added.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;performance&lt;/code&gt; suite (nightly at the time; weekly since this fix): &lt;code dir=&quot;auto&quot;&gt;tests/agent/glossary/test_pipeline_integration.py::TestPipelinePerformance::test_pipeline_performance_100_terms&lt;/code&gt; (4.295 s against 0.2 s) and &lt;code dir=&quot;auto&quot;&gt;tests/specify_cli/invocation/test_doctor_ops.py::test_sweep_nfr_002_10k_files_under_5s&lt;/code&gt; (5.088 s against 5.0 s), run 44230080019, #7704&lt;/td&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;&lt;strong&gt;No code regression; a latent order dependence fixed, then the statistic tuned with evidence.&lt;/strong&gt; Nothing on either path changed between the green run 44009807449 and the red one (the only &lt;code dir=&quot;auto&quot;&gt;src/kernel&lt;/code&gt; change, &lt;code dir=&quot;auto&quot;&gt;kernel/git/read_cache.py&lt;/code&gt;, is entered only by &lt;code dir=&quot;auto&quot;&gt;consolidate&lt;/code&gt;), and the runner image was the same. Eight green nightly runs put the glossary step at 0.059-0.103 s and the 10k sweep at 0.731-1.625 s; the red run took 1.76x longer overall (2008 s against 1143 s) with rmtree teardowns 12x slower, and the identical-workload &lt;code dir=&quot;auto&quot;&gt;test_hundred_watch_terms_under_200ms&lt;/code&gt; passed under 0.2 s minutes earlier in the same process, so one step stalled. Investigating it found a real defect: each glossary pipeline performance test timed the process’s first glossary event, which imports &lt;code dir=&quot;auto&quot;&gt;jsonschema&lt;/code&gt; lazily inside &lt;code dir=&quot;auto&quot;&gt;spec_kitty_events&lt;/code&gt;; in the dev environment that import costs about 1 s (0.9 s of it &lt;code dir=&quot;auto&quot;&gt;rfc3987_syntax&lt;/code&gt; building a Lark grammar, pulled in by the &lt;code dir=&quot;auto&quot;&gt;lint&lt;/code&gt; extra’s &lt;code dir=&quot;auto&quot;&gt;cyclonedx-bom[validation]&lt;/code&gt;; 0.11 s without it, so consumers are unaffected), and all five tests failed when run alone. By operator ruling the cost is now removed rather than budgeted: &lt;code dir=&quot;auto&quot;&gt;cyclonedx-bom&lt;/code&gt; moved to the non-default &lt;code dir=&quot;auto&quot;&gt;sbom&lt;/code&gt; dependency group that only &lt;code dir=&quot;auto&quot;&gt;release.yml&lt;/code&gt; installs. The dev &lt;code dir=&quot;auto&quot;&gt;--all-extras&lt;/code&gt; cold first step fell from 1.10-1.26 s to 0.11-0.16 s, the same as a consumer install (0.11-0.14 s), and &lt;code dir=&quot;auto&quot;&gt;tests/ci/test_dev_env_skips_rfc3987_syntax.py&lt;/code&gt; walks &lt;code dir=&quot;auto&quot;&gt;uv.lock&lt;/code&gt; to keep it out. They now take the median of &lt;code dir=&quot;auto&quot;&gt;IN_PROCESS_SAMPLE_RUNS&lt;/code&gt; (5) steps after one untimed warm-up, each on a fresh repository (&lt;code dir=&quot;auto&quot;&gt;median_in_process_seconds&lt;/code&gt; in &lt;code dir=&quot;auto&quot;&gt;tests/_perf_helpers.py&lt;/code&gt;), with the 0.2 s budget unchanged. The NFR-002 sweep keeps 5.0 s, the requirement itself (spec &lt;code dir=&quot;auto&quot;&gt;do-dispatch-open-op-lifecycle-01KTSJ2H&lt;/code&gt;; 3.1x the slowest clean figure), because absorbing the stall would put the number above the requirement; it takes the median of five sweeps over the same files instead. Every sample counts, so this is a fixed sampling design, not a retry; the operator accepted median-of-5 as tuning the gate, not a retry (ruling on #7710). Unit tests: with an injected clock they show one stalled call cannot decide the median and a uniformly slower call still moves it. The same ruling moved the &lt;code dir=&quot;auto&quot;&gt;performance&lt;/code&gt; job from nightly to weekly (Sunday cron in &lt;code dir=&quot;auto&quot;&gt;ci-nightly.yml&lt;/code&gt;, or a manual &lt;code dir=&quot;auto&quot;&gt;workflow_dispatch&lt;/code&gt;); its P0 escalation is unchanged, and &lt;code dir=&quot;auto&quot;&gt;tests/ci/test_performance_weekly_cadence.py&lt;/code&gt; pins the cadence. The release gate (&lt;code dir=&quot;auto&quot;&gt;scripts/ci/release_nightly_gate.py&lt;/code&gt;) now also requires a green run from the last 7 days in which the &lt;code dir=&quot;auto&quot;&gt;performance&lt;/code&gt; job executed, on the release commit or a &lt;code dir=&quot;auto&quot;&gt;main&lt;/code&gt; ancestor of it. No retry was added.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/specify_cli/decisions/test_service_terminal_concurrency.py&lt;/code&gt;’s coexistence ratchet (&lt;code dir=&quot;auto&quot;&gt;test_terminal_calls_do_not_deadlock_with_other_decision_lock_holders&lt;/code&gt;), #6773&lt;/td&gt;&lt;td&gt;2&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Fixed at root cause.&lt;/strong&gt; A correctness ratchet leaned on a wall clock it did not own: the product’s fixed 10 s decisions-lock acquire bound (then in &lt;code dir=&quot;auto&quot;&gt;decisions/service.py&lt;/code&gt;, mirrored in &lt;code dir=&quot;auto&quot;&gt;_decisions_doctor.py&lt;/code&gt;; since #6763 one constant, &lt;code dir=&quot;auto&quot;&gt;decisions/lock.py&lt;/code&gt;’s &lt;code dir=&quot;auto&quot;&gt;ACQUIRE_TIMEOUT_S&lt;/code&gt;). A holder that was merely slow surfaced as a &lt;code dir=&quot;auto&quot;&gt;DecisionError(LOCK_TIMEOUT)&lt;/code&gt; — a &lt;em&gt;retryable refusal by design&lt;/em&gt; (&lt;code dir=&quot;auto&quot;&gt;#4848&lt;/code&gt;) — and the harness reported it as a failed worker, with no deadlock present. There is no lock-order cycle among the four participants (each takes the decisions lock before the status lock, or only one of the two), the two short joins the report blamed are followed by &lt;code dir=&quot;auto&quot;&gt;assert is_alive()&lt;/code&gt; (load can only make those pass), and the surface did not reproduce on demand (7 broad runs / 2,107 executions, 14 file runs under &lt;code dir=&quot;auto&quot;&gt;-n 4&lt;/code&gt;, 40 in-process iterations, one run pinned to a single core — all green), and 30 consecutive &lt;code dir=&quot;auto&quot;&gt;-n 4 --dist loadfile&lt;/code&gt; runs of the fixed file were green while the shared host’s load average moved between 24 and 130 (individual runs took 8–269 s of wall clock). The scenario now pins that one acquire bound for every participant, one harness-owned &lt;code dir=&quot;auto&quot;&gt;_DEADLOCK_BUDGET_S&lt;/code&gt; decides “stuck”, and a timed-out join fails with a thread dump plus the decisions-lock holder record. No retry, nothing skipped or quarantined; the product’s refusal contract keeps its own coverage in &lt;code dir=&quot;auto&quot;&gt;test_decisions_lock_waiter_times_out_and_writes_nothing&lt;/code&gt;. Evidence: &lt;code dir=&quot;auto&quot;&gt;test_coexistence_ratchet_is_indifferent_to_the_ambient_acquire_bound&lt;/code&gt; goes red with &lt;code dir=&quot;auto&quot;&gt;DecisionError(LOCK_TIMEOUT)&lt;/code&gt; when the pin is removed, and &lt;code dir=&quot;auto&quot;&gt;test_deadlock_backstop_reports_a_participant_that_never_finishes&lt;/code&gt; proves a stuck participant still fails.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/specify_cli/shims/test_registry.py&lt;/code&gt; (parallel-collection nondeterminism)&lt;/td&gt;&lt;td&gt;2&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Fixed at root cause.&lt;/strong&gt; Parametrising over &lt;code dir=&quot;auto&quot;&gt;list(&amp;#x3C;frozenset&gt;)&lt;/code&gt; produced a &lt;code dir=&quot;auto&quot;&gt;PYTHONHASHSEED&lt;/code&gt;-dependent case order, so xdist workers collected different orders (“Different tests were collected between gw0 and gwN”). Changed to &lt;code dir=&quot;auto&quot;&gt;sorted(&amp;#x3C;frozenset&gt;)&lt;/code&gt;, making collection order deterministic across workers. Verified: identical node-id order under different hash seeds.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;tests/sync/tracker/test_egress_single_authority.py&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tests/sync/test_transport_revocation_matrix.py&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tests/sync/tracker/test_saas_client*.py&lt;/code&gt; and siblings (module-level accumulator / thread-identity state under bare &lt;code dir=&quot;auto&quot;&gt;-n &amp;#x3C;N&gt;&lt;/code&gt;)&lt;/td&gt;&lt;td&gt;2&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Retired — this surface no longer exists.&lt;/strong&gt; The sync transport that owned it was deleted (issue #5): &lt;code dir=&quot;auto&quot;&gt;tests/sync/&lt;/code&gt; is gone from the repo, so there is nothing left here to fix or re-flake. The root-cause fix this row used to describe (&lt;code dir=&quot;auto&quot;&gt;tests/conftest.py&lt;/code&gt;’s default &lt;code dir=&quot;auto&quot;&gt;--dist loadfile&lt;/code&gt; promotion for xdist) is unaffected — it applies to any file, not specifically to the deleted &lt;code dir=&quot;auto&quot;&gt;tests/sync/&lt;/code&gt; tree.&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Historical &lt;code dir=&quot;auto&quot;&gt;tests/sync/test_daemon_self_retirement.py&lt;/code&gt; daemon-self-retirement reports&lt;/td&gt;&lt;td&gt;2&lt;/td&gt;&lt;td&gt;&lt;strong&gt;Retired — this surface no longer exists.&lt;/strong&gt; The sync transport that owned it was deleted (issue #5); &lt;code dir=&quot;auto&quot;&gt;tests/sync/test_daemon_self_retirement.py&lt;/code&gt; and its old integration receipt are gone from the repo.&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;h2 id=&quot;adding-a-new-test-avoid-the-common-root-causes&quot;&gt;Adding a new test? Avoid the common root causes&lt;/h2&gt;
&lt;p&gt;When a correctness (Tier 2) test flakes, it is almost always one of these — fix
the cause, do not retry:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Unordered data used where order matters.&lt;/strong&gt; Iterating a &lt;code dir=&quot;auto&quot;&gt;set&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;frozenset&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;dict&lt;/code&gt;
for parametrize IDs or assertion sequences → &lt;code dir=&quot;auto&quot;&gt;PYTHONHASHSEED&lt;/code&gt;-dependent order.
Wrap in &lt;code dir=&quot;auto&quot;&gt;sorted(...)&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Fixture-teardown races / leaked global state&lt;/strong&gt; between tests sharing a module
or process.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;monkeypatch&lt;/code&gt; / env-var leakage&lt;/strong&gt; across tests (rely on the per-worker HOME
isolation; don’t mutate process-global state without restoring it).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Import-time side effects&lt;/strong&gt; that bind a path or singleton before fixtures run.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Time-sensitive assertions&lt;/strong&gt; in a non-timing test — move the timing concern to
a Tier-1 budget gate with a generous threshold, or remove the wall-clock
dependency.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;patch.dict(sys.modules, ...)&lt;/code&gt;&lt;/strong&gt; — snapshot-restores the whole dict on exit,
evicting any module first-imported inside the patched window (spec-kitty#89/#99;
ratcheted by &lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_no_sys_modules_patch_dict.py&lt;/code&gt;). Use
&lt;code dir=&quot;auto&quot;&gt;monkeypatch.setitem(sys.modules, key, value)&lt;/code&gt; per key instead — it is the only
accepted &lt;code dir=&quot;auto&quot;&gt;sys.modules&lt;/code&gt; seam.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;test-run-baseline-red-gotcha&quot;&gt;Test-run baseline-red gotcha&lt;/h2&gt;
&lt;p&gt;A red test is not automatically &lt;em&gt;your&lt;/em&gt; red. A local or backgrounded &lt;code dir=&quot;auto&quot;&gt;pytest&lt;/code&gt; run over
anything broad (the full suite, &lt;code dir=&quot;auto&quot;&gt;tests/merge/&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tests/architectural/&lt;/code&gt;, the regression job)
will surface failures you did not cause. &lt;strong&gt;Classify every failure before you act on it&lt;/strong&gt; —
misattribution wastes effort and, worse, tempts an agent to green-wash a signal the project
deliberately keeps red.&lt;/p&gt;
&lt;p&gt;Six baseline-red categories that are &lt;strong&gt;not yours to fix&lt;/strong&gt;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Pre-existing known-P0 reds.&lt;/strong&gt; Per &lt;a href=&quot;/adr/3.x/2026-07-17-1-red-main-is-honest-ci-is-release-authority/&quot;&gt;ADR 2026-07-17-1&lt;/a&gt;,
an open P0 bug is &lt;em&gt;expected&lt;/em&gt; to red mainline. Its reproduction carries
&lt;code dir=&quot;auto&quot;&gt;@pytest.mark.p0_repro(issue=N)&lt;/code&gt; and runs only in the nightly &lt;code dir=&quot;auto&quot;&gt;p0-repro&lt;/code&gt; lane, so it
never reds a PR. You will see it only in a nightly run or in an explicit
&lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_RUN_P0_REPRO=1&lt;/code&gt; run, where each failure names its &lt;code dir=&quot;auto&quot;&gt;[OPEN P0 #N]&lt;/code&gt; issue.
&lt;strong&gt;Leave it red.&lt;/strong&gt; Do not unmark it, quarantine it, or “fix” it in an unrelated change
(the fold-first policy in &lt;a href=&quot;/development/how-to/pr-landing/&quot;&gt;pr-landing.md&lt;/a&gt;).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;CI-environment failures.&lt;/strong&gt; Auth state (&lt;code dir=&quot;auto&quot;&gt;logged_out_on_connected_teamspace&lt;/code&gt; during
&lt;code dir=&quot;auto&quot;&gt;upgrade&lt;/code&gt;) and the gate opt-out &lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_SKIP_PRE_REVIEW_GATE&lt;/code&gt; (the pre-review
gate’s own name since #3980 — it no longer reads the sync-disable vocabulary) make
some CI jobs red while the same tests pass locally. These are configuration, not
your diff.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Stale-install false reds.&lt;/strong&gt; Product code that shells out to &lt;code dir=&quot;auto&quot;&gt;spec-kitty&lt;/code&gt; (e.g. the
&lt;code dir=&quot;auto&quot;&gt;merge-driver-meta&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;-traces&lt;/code&gt; commands) only fires when an up-to-date &lt;code dir=&quot;auto&quot;&gt;spec-kitty&lt;/code&gt; is
installed. Between landing a change and &lt;code dir=&quot;auto&quot;&gt;pip install -e .&lt;/code&gt;, coverage/gate jobs report
false reds for lines that are actually exercised via subprocess.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Stale-venv false reds.&lt;/strong&gt; A &lt;code dir=&quot;auto&quot;&gt;ModuleNotFoundError&lt;/code&gt; (or other import failure) for a package
that &lt;em&gt;is&lt;/em&gt; declared and pinned (&lt;code dir=&quot;auto&quot;&gt;pyproject.toml&lt;/code&gt; / &lt;code dir=&quot;auto&quot;&gt;uv.lock&lt;/code&gt;) usually means the local
&lt;code dir=&quot;auto&quot;&gt;.venv&lt;/code&gt; was never (re)synced to that pin, not a real regression. Run
&lt;code dir=&quot;auto&quot;&gt;uv sync --frozen --all-extras&lt;/code&gt; and retry the failing test before recording it as
pre-existing or unrelated — a stale venv is indistinguishable from real breakage in raw
pytest output (#648: a PR’s &lt;code dir=&quot;auto&quot;&gt;## Tests run&lt;/code&gt; section excluded a whole
test file over exactly this &lt;code dir=&quot;auto&quot;&gt;ModuleNotFoundError&lt;/code&gt;, when a clean &lt;code dir=&quot;auto&quot;&gt;uv sync --frozen --all-extras&lt;/code&gt; reproduced 1621/1621 passing with no exclusion needed).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;A git checkout above basetemp.&lt;/strong&gt; When pytest’s basetemp has an ancestor that is
itself a git work tree (every exe.dev VM, an agent sandbox with &lt;code dir=&quot;auto&quot;&gt;TMPDIR&lt;/code&gt; under a
checkout, a &lt;code dir=&quot;auto&quot;&gt;/tmp&lt;/code&gt; someone ran &lt;code dir=&quot;auto&quot;&gt;git init&lt;/code&gt; in), git discovery from a &lt;code dir=&quot;auto&quot;&gt;tmp_path&lt;/code&gt; climbs
into that foreign checkout: status and lifecycle writers land their lock inside its
&lt;code dir=&quot;auto&quot;&gt;.git&lt;/code&gt; (if it is read-only, &lt;code dir=&quot;auto&quot;&gt;append_lifecycle_event&lt;/code&gt; swallows the &lt;code dir=&quot;auto&quot;&gt;PermissionError&lt;/code&gt; and
persists nothing, 130 nodes red in the #142 repro), mission-state repair re-anchors on
it and finds no missions, and the non-repository git probe is never reached (18
fast-tier nodes red in #7022 and #5356, CI green). This is host setup, not your diff.
The guard is run-wide: &lt;code dir=&quot;auto&quot;&gt;tests/conftest.py&lt;/code&gt; appends the run’s basetemp and the worker’s
isolated home to &lt;code dir=&quot;auto&quot;&gt;GIT_CEILING_DIRECTORIES&lt;/code&gt; (&lt;code dir=&quot;auto&quot;&gt;tests/_support/git_ceiling.py&lt;/code&gt;), which git
and the root walks in &lt;code dir=&quot;auto&quot;&gt;specify_cli.core.paths&lt;/code&gt; honor. It is pinned end to end by
&lt;code dir=&quot;auto&quot;&gt;tests/unit/test_harness_git_ceiling.py&lt;/code&gt; (temp root inside an outer repository) and
&lt;code dir=&quot;auto&quot;&gt;tests/status/test_hermetic_ambient_git.py&lt;/code&gt; (read-only ancestor &lt;code dir=&quot;auto&quot;&gt;.git&lt;/code&gt;, non-root
hosts). A test that builds its own environment from scratch, or a directory made with
&lt;code dir=&quot;auto&quot;&gt;tempfile&lt;/code&gt; outside basetemp, does not inherit the ceiling; set
&lt;code dir=&quot;auto&quot;&gt;GIT_CEILING_DIRECTORIES&lt;/code&gt; there yourself.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Harness environment leaks.&lt;/strong&gt; Agent containers export &lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_NO_NAG&lt;/code&gt;,
&lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_NO_UPGRADE_CHECK&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_NON_INTERACTIVE&lt;/code&gt; into the shell that runs
pytest. The root test fixture keeps pre-existing &lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_*&lt;/code&gt; values by design, so a
test that needs the behaviour such a variable switches off (for example the upgrade
nag) goes red only inside an agent container and passes in CI. The remedy is a
test-local &lt;code dir=&quot;auto&quot;&gt;monkeypatch.delenv(&quot;&amp;#x3C;NAME&gt;&quot;, raising=False)&lt;/code&gt;, never an edit of the root
fixture and never a skip.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;strong&gt;The attribution test:&lt;/strong&gt; a failure is yours to fold only if it is &lt;strong&gt;red on your branch and
green on the base&lt;/strong&gt;. Confirm the base state by running the same node id against
&lt;code dir=&quot;auto&quot;&gt;upstream/main&lt;/code&gt; — e.g. from a throwaway worktree with
&lt;code dir=&quot;auto&quot;&gt;PYTHONPATH=&quot;$(pwd)/src&quot; python -m pytest &amp;#x3C;nodeid&gt;&lt;/code&gt; — or by checking the tracker for a P0
label. When you &lt;em&gt;do&lt;/em&gt; add a red-first P0 reproduction on purpose (per the ADR), mark it
&lt;code dir=&quot;auto&quot;&gt;p0_repro(issue=N)&lt;/code&gt;, docstring the issue, and make sure it fails for the product reason,
not because of setup. It runs only in the nightly; see
&lt;a href=&quot;/development/reference/red-main-and-release-readiness/#where-the-reproduction-runs-the-nightly-p0-repro-lane&quot;&gt;Red Main and Release Readiness&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;This applies to &lt;strong&gt;dispatched subagents&lt;/strong&gt; as much as the orchestrator: an implementer that runs
the suite in its worktree must not report the baseline reds as regressions or try to fix them.&lt;/p&gt;
&lt;h2 id=&quot;see-also&quot;&gt;See also&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;/development/testing/testing-parallel/&quot;&gt;Running the test suite in parallel&lt;/a&gt; — per-worker HOME
isolation, the serial marker passes, and the stability ratchet.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;/adr/3.x/2026-04-20-1-mutation-testing-as-local-only-quality-gate/&quot;&gt;ADR 2026-04-20-1&lt;/a&gt;
— the &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; / &lt;code dir=&quot;auto&quot;&gt;non_sandbox&lt;/code&gt; markers (mutmut deselection; distinct from this
policy).&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;/development/testing/run-mutation-tests/&quot;&gt;How-to: run mutation tests&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;</content>
</entry>
