Skip to content

Test-flakiness handling policy

Who is this for (1)

A flaky test is one whose pass/fail outcome changes between runs without any change to the code under test — it sometimes goes red on CI for reasons unrelated to the diff under review. Flakes waste review cycles and, left unmanaged, normalise a red CI that everyone learns to ignore.

This page is the suite-wide policy for handling them: how we detect a flake, what we are allowed to do about it (and explicitly not allowed to do), and the current disposition of every known flake surface in this repo.

The one rule that governs everything below: we never make a test pass by retrying it until it goes green. A green-after-retry is the “fixed because it looks fixed” trap — it hides genuine regressions and is incompatible with this repo’s live-evidence / anti-laziness discipline. Retry plugins (pytest-rerunfailures, the PyPI flaky plugin) are therefore not adopted, and are not present in the dependency set.

⚠️ Naming: flaky the marker is not “flaky” the concept

This repo already registers a flaky pytest marker, and it means something narrow and unrelated to this policy. Per ADR 2026-04-20-1, @pytest.mark.flaky means “passes reliably in the main suite but is non-deterministic under mutmut / forking pipelines” — it is a deselection bucket for the mutation-testing sandbox, not a CI-quarantine mechanism.

Do not reach for the flaky marker to deal with a CI flake. If a genuine quarantine mechanism is ever needed, it must use a different name (quarantine, see below) so the two never get conflated.

The three tiers

Every flake in this suite falls into one of three tiers, and each tier has a single sanctioned response.

TierWhat it isExampleSanctioned response
1. Threshold / budget gateA test that asserts a measurement stays under a wall-clock / size budget. CI runners are shared and noisy, so a generous gate occasionally trips with no real regression.Timing-budget tests (tests/architectural/test_spec_kitty_home_pin_budget.py), the -m timing gate.Tune the budget — never retry. Tune a budget only with evidence: a real regression adds time consistently and still trips a calibrated gate; a retry would mask exactly that. A test that pays interpreter cold start (a fresh python -m specify_cli per run) does not take an absolute number of seconds at all: it asserts a runner-relative measure, the ratio of its median to a start-up floor median sampled interleaved in the same run, with the limit in tests/_perf_helpers.py and calibrated from recorded data (see the audit row and the decision record). Widening an absolute budget to absorb a slow runner is not accepted for these tests.
2. Correctness testA test of logical behaviour that should be 100% deterministic. If it flakes, the test (or the code) has a hidden nondeterminism — shared global state, ordering assumptions, fixture-teardown races, monkeypatch leakage, import-time side effects.tests/specify_cli/shims/test_registry.py (parallel-collection nondeterminism).Fix the root cause — never retry. A correctness test that needs a retry is lying. Find the nondeterminism and remove it.
3. Genuinely environmentalA test that depends on an OS-global resource — real TCP ports, singleton daemons, the real filesystem — that cannot be fully isolated per worker.Historical real-port / daemon suites, deleted with the sync transport (issue #5).Surgical handling only. First serialise (-n0) and isolate (per-worker HOME) — see testing-parallel.md. Only if a residual, irreducible environmental flake remains do we quarantine (below) — never a blanket retry.

Detection: confirm a flake before you treat it

One red run is a single data point — pytest alone cannot tell flaky from broken. Before declaring a test “flaky” and applying any of the responses above, reproduce the nondeterminism:

  • Re-run the test in isolation, and under the parallel runner it failed on:
    Terminal window
    PWHEADLESS=1 pytest <path> -n auto --dist loadfile -p no:cacheprovider
  • For ordering / hash-seed flakes, re-run under different PYTHONHASHSEED values and diff the collected node IDs.
  • For a confidence signal, use the existing stability ratchet — N consecutive green parallel runs, the same gate CI uses for shard flips. The authoritative invocation lives in testing-parallel.md → Running the stability ratchet locally; point it at the suspect <path> instead of fabricating a new command. (Full harness: tests/_support/coverage_safety/README.md.)

A test that cannot be made to fail again under these probes is not confirmed flaky — do not annotate it.

Tooling decision

No retry tooling — ever. pytest-rerunfailures / PyPI flaky are deliberately not in the dependency set (see the rule at the top). Detection and isolation reuse what the repo already has; the only thing built specifically for this policy is the quarantine marker.

  • No retry plugin. See the rule at the top.
  • Detection uses the existing stability ratchet (tests._support.coverage_safety.ratchet), not a new confidence tool.
  • Isolation (the first line of defence for Tier 3) is the existing per-worker HOME isolation + serial -n0 pass documented in testing-parallel.md.

quarantine — built, env-gated, non-blocking

The sanctioned mechanism for an irreducible Tier-3 flake is a dedicated quarantine marker — not a retry, and not the existing flaky marker. It is implemented as a single, un-bypassable chokepoint:

  1. Registered canonically in pytest.ini’s markers block — the single source of truth for the marker registry (#2034) — sufficient for --strict-markers.
  2. Held out of every normal run. tests/conftest.py’s pytest_collection_modifyitems skips any @pytest.mark.quarantine test unless SPEC_KITTY_RUN_QUARANTINE=1. Because this is collection-time and global, no -m selector in any CI job (present or future) can accidentally run a quarantined test — the gate cannot be forgotten. The opt-in is strict (only the literal "1"); the pure decision lives in tests/_support/quarantine.py and is unit-tested.
  3. Visible only by explicit opt-in; not currently scheduled in CI. Setting SPEC_KITTY_RUN_QUARANTINE=1 and selecting -m quarantine runs a quarantined test for real locally. The current hosted CI topology has one suite job and no quarantine-visibility lane, so quarantine visibility is not enforced there. A PR that adds the first quarantined test must also add a non-blocking visibility lane (or explicitly change this policy); it must not rely on the normal suite, which skips quarantined tests.

To quarantine a test: mark it @pytest.mark.quarantine with a one-line reason and a tracking-issue link — every quarantined test is tech debt with an owner. The wiring above (test_quarantine_marker.py) is enforced.

As of this writing no test is quarantined (see the disposition table — every known surface is fixed, correctly handled, or retired). The mechanism exists so the first irreducible flake has a sanctioned home instead of a retry.

Audit + disposition of known flake surfaces

SurfaceTierDisposition
tests/charter/activation/test_stored_spelling_6527.py::test_a_writer_that_names_only_in_force_artifacts_stores_nothing[cascade] (before #6685: test_writer_stores_stems_only[cascade]; over the 240 s per-test backstop)1Root cause partly fixed; the interim budget stays until #6854 (#6777). The case is not a budget test: it drives a real charter activate --cascade all, and its cost is avoidable work. Part one is fixed: the cascade resolved each of its 114 targets with its own fresh identity pass, re-walking and re-parsing the whole artifact catalog per target (121 lookups, 5309 id reads, 5028 artifact-file parses); one shared ResolutionPass covering the whole batch before the first write takes it to 240 parses and 0.62–0.70× the wall clock (interleaved A/B; absolute seconds on a shared host are not evidence). Part two remains: each target’s ActiveCharterManager.activate() still rescans its kind’s catalog (114 scans where 6 would do), and with only part one fixed the case still crossed 240 s on a per-PR runner (run 42462186463). So #6776’s 600 s per-test budget (_CASCADE_TIMEOUT_S) is kept as a budget-gate tune, not a retry, beside the documented slow tier the module shards still select. Two guards pin it: one fails if any other case overrides the backstop or if the budget’s comment stops naming #6854 (so dropping it is a deliberate act when #6854 lands), and one fails if the batch ever stops sharing one pass or resolves after the first write. No retry was added.
Historical tests/architectural/test_wp_prompt_build_latency.py NFR-002 latency surface1Retired — this surface no longer exists. The low-signal suite cleanup in PR #3285 removed the file. The current timing policy is carried by the live -m timing lane and its current budget tests; a future latency flake gets a new, separately evidenced row.
doctor restart-daemon issue #1153 / NFR-002 wall-clock gate3Retired as a portable CI performance claim in PR #3285. The original ≤10-second end-to-end requirement depended on OS-global daemon and port state and repeatedly failed during hosted-macOS control-plane bootstrap without a corresponding product regression. The controlled Linux lane now enforces the replacement functional contract: the old daemon stops, a distinct PID starts, and the control plane becomes healthy within bounded subprocess and readiness timeouts. macOS remains supported through platform-independent tests and local smoke evidence; a future cross-platform restart performance SLO requires a controlled developer/canary harness because shared-runner wall clock is not accepted as evidence.
tests/sync/test_orphan_sweep.py and tests/_real_port_suites.py’s FIXED_RANGE_SUITES (real fixed ports 9400–9449, daemons)3Retired — this surface no longer exists. The sync transport that owned it was deleted (issue #5): tests/sync/ and tests/_real_port_suites.py are gone from the repo, so there is nothing left to serialise or quarantine here. make test-full’s current topology (one parallel pass plus dedicated -n0 serial passes for stress/timing) is described in CLAUDE.md’s Commands section.
tests/performance/test_owned_checkout_perf.py (three owned-checkout commands), test_cli_startup_budget_4409.py (--help start-up) and test_cli_startup_agent_commands_freshness.py (warm repeated leaf command), nightly performance job, #5419 / #56141Fixed by a runner-relative measure. The three owned-checkout tests compared a median with an absolute 2.5 s and went red on four of six nightly runs with no product change: the same bare --help moved from 1.14 s to 1.95 s between runs. They now assert the ratio of the command median to a start-up floor median (--version), five spawns of each interleaved in the same run, against OWNED_CHECKOUT_RATIO_LIMIT; the two start-up tests assert the ratio of the start-up median to a fixed interpreter workload measured in the same run (STARTUP_RATIO_LIMIT, WARM_LEAF_RATIO_LIMIT). No absolute number of seconds remains. Every limit lives in tests/_perf_helpers.py and tests/architectural/test_perf_limit_authority.py fails when another test file defines one. The limits come from recorded clean and planted ratios (idle and CPU-throttled), with their headroom, in the decision record; committed planted-work tests prove each relative assertion can fail. No retry was added.
tests/architectural/test_same_tier_uniqueness.py universe fixture (16 Failed: Timeout (>240.0s) errors in the architectural battery 2/2 leg, #7052)1Fixed by tuning the budget, with evidence. The module-scoped universe fixture reuses the stored test-universe record (0.6 s) but collects the whole tree itself when the store is bypassed (a dirty checkout, no record): 89-233 s for a first CI run, 247.7 s on the reporting machine, 90-422 s on a shared development VM. That real collection sat at or over the suite’s 240 s per-test backstop, so it was killed and every test depending on the fixture errored at once. _gate_coverage.COLLECT_TIMEOUT_S now names the collection ceiling once and the tests whose fixture closure reaches the universe carry @pytest.mark.timeout(COLLECT_TIMEOUT_S + 300); the suite-wide 240 s default is unchanged, a test that never collects keeps it, and a guard test fails if a consumer’s budget is missing or below the ceiling or if a collection subprocess repeats the number as a literal. No retry was added, and a bypass on CI is still a check-step failure with its reason and dirty paths. See ci-gate-mechanics.
Nightly acceptance-bdd scenario “Consolidating lands the approved work and leaves my untracked files alone” (tests/scenarios/consolidate/consolidate.feature, CliTimeoutException after the driver’s 2-minute DEFAULT_TIMEOUT on windows-latest, run 44009807449), #76891Root cause partly fixed, then the budget tuned with evidence. Not a hang: the consolidate process itself was still running at 120 s (the waitFor branch, not the pipe drain), no stdin prompt is reachable with --yes, and no commit in the green-to-red window changed its work (671 git processes at both SHAs, 15-21 s on Linux). The red run’s own cucumber.json shows that Windows runner 7-9x slower than Linux on every CLI step, which puts consolidate at about 105-165 s. Part one is fixed: a run-scoped git read cache (kernel.git.read_cache, entered only by consolidate) answers repeated reads without a process while the files they depend on are unchanged, taking the same run from 671 to 470 git processes; tests/integration/test_7689_consolidate_git_spawn_ceiling.py keeps it at or below 500. That alone gives no safe margin on a slow Windows runner, so CliProcessDriver.CONSOLIDATE_TIMEOUT (4 minutes, about 1.5x the worst estimate) replaces the shared 2-minute ceiling for spec-kitty consolidate only; every other command keeps DEFAULT_TIMEOUT. tests/ci/test_acceptance_cli_ceilings.py fails if the default moves, if a named ceiling is not above it or exceeds MAX_COMMAND_TIMEOUT (at most 5 minutes), if another command gains a named ceiling, or if a ceiling’s comment or this row stops naming its issue. No retry was added.
performance suite (nightly at the time; weekly since this fix): tests/agent/glossary/test_pipeline_integration.py::TestPipelinePerformance::test_pipeline_performance_100_terms (4.295 s against 0.2 s) and tests/specify_cli/invocation/test_doctor_ops.py::test_sweep_nfr_002_10k_files_under_5s (5.088 s against 5.0 s), run 44230080019, #77041No code regression; a latent order dependence fixed, then the statistic tuned with evidence. Nothing on either path changed between the green run 44009807449 and the red one (the only src/kernel change, kernel/git/read_cache.py, is entered only by consolidate), and the runner image was the same. Eight green nightly runs put the glossary step at 0.059-0.103 s and the 10k sweep at 0.731-1.625 s; the red run took 1.76x longer overall (2008 s against 1143 s) with rmtree teardowns 12x slower, and the identical-workload test_hundred_watch_terms_under_200ms passed under 0.2 s minutes earlier in the same process, so one step stalled. Investigating it found a real defect: each glossary pipeline performance test timed the process’s first glossary event, which imports jsonschema lazily inside spec_kitty_events; in the dev environment that import costs about 1 s (0.9 s of it rfc3987_syntax building a Lark grammar, pulled in by the lint extra’s cyclonedx-bom[validation]; 0.11 s without it, so consumers are unaffected), and all five tests failed when run alone. By operator ruling the cost is now removed rather than budgeted: cyclonedx-bom moved to the non-default sbom dependency group that only release.yml installs. The dev --all-extras cold first step fell from 1.10-1.26 s to 0.11-0.16 s, the same as a consumer install (0.11-0.14 s), and tests/ci/test_dev_env_skips_rfc3987_syntax.py walks uv.lock to keep it out. They now take the median of IN_PROCESS_SAMPLE_RUNS (5) steps after one untimed warm-up, each on a fresh repository (median_in_process_seconds in tests/_perf_helpers.py), with the 0.2 s budget unchanged. The NFR-002 sweep keeps 5.0 s, the requirement itself (spec do-dispatch-open-op-lifecycle-01KTSJ2H; 3.1x the slowest clean figure), because absorbing the stall would put the number above the requirement; it takes the median of five sweeps over the same files instead. Every sample counts, so this is a fixed sampling design, not a retry; the operator accepted median-of-5 as tuning the gate, not a retry (ruling on #7710). Unit tests: with an injected clock they show one stalled call cannot decide the median and a uniformly slower call still moves it. The same ruling moved the performance job from nightly to weekly (Sunday cron in ci-nightly.yml, or a manual workflow_dispatch); its P0 escalation is unchanged, and tests/ci/test_performance_weekly_cadence.py pins the cadence. The release gate (scripts/ci/release_nightly_gate.py) now also requires a green run from the last 7 days in which the performance job executed, on the release commit or a main ancestor of it. No retry was added.
tests/specify_cli/decisions/test_service_terminal_concurrency.py’s coexistence ratchet (test_terminal_calls_do_not_deadlock_with_other_decision_lock_holders), #67732Fixed at root cause. A correctness ratchet leaned on a wall clock it did not own: the product’s fixed 10 s decisions-lock acquire bound (then in decisions/service.py, mirrored in _decisions_doctor.py; since #6763 one constant, decisions/lock.py’s ACQUIRE_TIMEOUT_S). A holder that was merely slow surfaced as a DecisionError(LOCK_TIMEOUT) — a retryable refusal by design (#4848) — and the harness reported it as a failed worker, with no deadlock present. There is no lock-order cycle among the four participants (each takes the decisions lock before the status lock, or only one of the two), the two short joins the report blamed are followed by assert is_alive() (load can only make those pass), and the surface did not reproduce on demand (7 broad runs / 2,107 executions, 14 file runs under -n 4, 40 in-process iterations, one run pinned to a single core — all green), and 30 consecutive -n 4 --dist loadfile runs of the fixed file were green while the shared host’s load average moved between 24 and 130 (individual runs took 8–269 s of wall clock). The scenario now pins that one acquire bound for every participant, one harness-owned _DEADLOCK_BUDGET_S decides “stuck”, and a timed-out join fails with a thread dump plus the decisions-lock holder record. No retry, nothing skipped or quarantined; the product’s refusal contract keeps its own coverage in test_decisions_lock_waiter_times_out_and_writes_nothing. Evidence: test_coexistence_ratchet_is_indifferent_to_the_ambient_acquire_bound goes red with DecisionError(LOCK_TIMEOUT) when the pin is removed, and test_deadlock_backstop_reports_a_participant_that_never_finishes proves a stuck participant still fails.
tests/specify_cli/shims/test_registry.py (parallel-collection nondeterminism)2Fixed at root cause. Parametrising over list(<frozenset>) produced a PYTHONHASHSEED-dependent case order, so xdist workers collected different orders (“Different tests were collected between gw0 and gwN”). Changed to sorted(<frozenset>), making collection order deterministic across workers. Verified: identical node-id order under different hash seeds.
tests/sync/tracker/test_egress_single_authority.py, tests/sync/test_transport_revocation_matrix.py, tests/sync/tracker/test_saas_client*.py and siblings (module-level accumulator / thread-identity state under bare -n <N>)2Retired — this surface no longer exists. The sync transport that owned it was deleted (issue #5): tests/sync/ is gone from the repo, so there is nothing left here to fix or re-flake. The root-cause fix this row used to describe (tests/conftest.py’s default --dist loadfile promotion for xdist) is unaffected — it applies to any file, not specifically to the deleted tests/sync/ tree.
Historical tests/sync/test_daemon_self_retirement.py daemon-self-retirement reports2Retired — this surface no longer exists. The sync transport that owned it was deleted (issue #5); tests/sync/test_daemon_self_retirement.py and its old integration receipt are gone from the repo.

Adding a new test? Avoid the common root causes

When a correctness (Tier 2) test flakes, it is almost always one of these — fix the cause, do not retry:

  • Unordered data used where order matters. Iterating a set/frozenset/dict for parametrize IDs or assertion sequences → PYTHONHASHSEED-dependent order. Wrap in sorted(...).
  • Fixture-teardown races / leaked global state between tests sharing a module or process.
  • monkeypatch / env-var leakage across tests (rely on the per-worker HOME isolation; don’t mutate process-global state without restoring it).
  • Import-time side effects that bind a path or singleton before fixtures run.
  • Time-sensitive assertions in a non-timing test — move the timing concern to a Tier-1 budget gate with a generous threshold, or remove the wall-clock dependency.
  • patch.dict(sys.modules, ...) — snapshot-restores the whole dict on exit, evicting any module first-imported inside the patched window (spec-kitty#89/#99; ratcheted by tests/architectural/test_no_sys_modules_patch_dict.py). Use monkeypatch.setitem(sys.modules, key, value) per key instead — it is the only accepted sys.modules seam.

Test-run baseline-red gotcha

A red test is not automatically your red. A local or backgrounded pytest run over anything broad (the full suite, tests/merge/, tests/architectural/, the regression job) will surface failures you did not cause. Classify every failure before you act on it — misattribution wastes effort and, worse, tempts an agent to green-wash a signal the project deliberately keeps red.

Six baseline-red categories that are not yours to fix:

  1. Pre-existing known-P0 reds. Per ADR 2026-07-17-1, an open P0 bug is expected to red mainline. Its reproduction carries @pytest.mark.p0_repro(issue=N) and runs only in the nightly p0-repro lane, so it never reds a PR. You will see it only in a nightly run or in an explicit SPEC_KITTY_RUN_P0_REPRO=1 run, where each failure names its [OPEN P0 #N] issue. Leave it red. Do not unmark it, quarantine it, or “fix” it in an unrelated change (the fold-first policy in pr-landing.md).

  2. CI-environment failures. Auth state (logged_out_on_connected_teamspace during upgrade) and the gate opt-out SPEC_KITTY_SKIP_PRE_REVIEW_GATE (the pre-review gate’s own name since #3980 — it no longer reads the sync-disable vocabulary) make some CI jobs red while the same tests pass locally. These are configuration, not your diff.

  3. Stale-install false reds. Product code that shells out to spec-kitty (e.g. the merge-driver-meta/-traces commands) only fires when an up-to-date spec-kitty is installed. Between landing a change and pip install -e ., coverage/gate jobs report false reds for lines that are actually exercised via subprocess.

  4. Stale-venv false reds. A ModuleNotFoundError (or other import failure) for a package that is declared and pinned (pyproject.toml / uv.lock) usually means the local .venv was never (re)synced to that pin, not a real regression. Run uv sync --frozen --all-extras and retry the failing test before recording it as pre-existing or unrelated — a stale venv is indistinguishable from real breakage in raw pytest output (#648: a PR’s ## Tests run section excluded a whole test file over exactly this ModuleNotFoundError, when a clean uv sync --frozen --all-extras reproduced 1621/1621 passing with no exclusion needed).

  5. A git checkout above basetemp. When pytest’s basetemp has an ancestor that is itself a git work tree (every exe.dev VM, an agent sandbox with TMPDIR under a checkout, a /tmp someone ran git init in), git discovery from a tmp_path climbs into that foreign checkout: status and lifecycle writers land their lock inside its .git (if it is read-only, append_lifecycle_event swallows the PermissionError and persists nothing, 130 nodes red in the #142 repro), mission-state repair re-anchors on it and finds no missions, and the non-repository git probe is never reached (18 fast-tier nodes red in #7022 and #5356, CI green). This is host setup, not your diff. The guard is run-wide: tests/conftest.py appends the run’s basetemp and the worker’s isolated home to GIT_CEILING_DIRECTORIES (tests/_support/git_ceiling.py), which git and the root walks in specify_cli.core.paths honor. It is pinned end to end by tests/unit/test_harness_git_ceiling.py (temp root inside an outer repository) and tests/status/test_hermetic_ambient_git.py (read-only ancestor .git, non-root hosts). A test that builds its own environment from scratch, or a directory made with tempfile outside basetemp, does not inherit the ceiling; set GIT_CEILING_DIRECTORIES there yourself.

  6. Harness environment leaks. Agent containers export SPEC_KITTY_NO_NAG, SPEC_KITTY_NO_UPGRADE_CHECK and SPEC_KITTY_NON_INTERACTIVE into the shell that runs pytest. The root test fixture keeps pre-existing SPEC_KITTY_* values by design, so a test that needs the behaviour such a variable switches off (for example the upgrade nag) goes red only inside an agent container and passes in CI. The remedy is a test-local monkeypatch.delenv("<NAME>", raising=False), never an edit of the root fixture and never a skip.

The attribution test: a failure is yours to fold only if it is red on your branch and green on the base. Confirm the base state by running the same node id against upstream/main — e.g. from a throwaway worktree with PYTHONPATH="$(pwd)/src" python -m pytest <nodeid> — or by checking the tracker for a P0 label. When you do add a red-first P0 reproduction on purpose (per the ADR), mark it p0_repro(issue=N), docstring the issue, and make sure it fails for the product reason, not because of setup. It runs only in the nightly; see Red Main and Release Readiness.

This applies to dispatched subagents as much as the orchestrator: an implementer that runs the suite in its worktree must not report the baseline reds as regressions or try to fix them.

See also