ADR: a green main/nightly must mean the tests actually ran
Context and Problem Statement
The per-module CI selector (scripts/ci/gate_selection.py, fed by ci-router.yml)
keyed which module shards run on a push to source roots — a change under
src/specify_cli/<x>/** selected the <x> module — with no model of which
source a given test exercises. Two blind spots followed:
- Cross-cutting / unenrolled suites were silently skipped on
mainpushes. Whole test directories (tests/agent,tests/specify_cli/live_work,tests/review,tests/upgrade, and theexecution_contextfamily) were effectively never selected on ordinarymainpushes, so a regression in code they cover could ship to a greenmainundetected. integration_tests_nextwas a dead tier — declared in the registry with rootstests/integration/**+tests/next/**but invoked by no workflow, so those suites ran nowhere.
This was root-caused in #5034 (epic #4437) after it repeatedly cost maintainer landing passes: PRs #5029 and #5032 each un-masked a backlog of pre-existing reds the moment their diff touched a path that finally selected one of those shards. A green check no longer meant the tests had run.
Decision
Make “green ⇒ the tests actually ran” an enforced property, in four parts:
- Coverage-honesty guards (
scripts/ci/coverage_guard_lib.py, enforced bytests/architectural/test_foreign_coverage_guard.pyandtest_src_reachability_guard.py) — shrink-only ratchets over a measured baseline (.github/ci-foreign-coverage-baseline.json). foreign-coverage requires every registry row’s test dirs to exercise its ownroots:; src-reachability pins atruly-darkset (a package imported by no test anywhere) and anin-matrix-darkset (tested only nightly). They fail only when coverage regresses, never vacuously. - Enrol the dark suites into
.github/ci-module-registry.yml(agent_utils,live_work,config,calibration,tasks_authoring,bootstrap) with re-measured host shard timings, and fix the phantom-mirror authority bug intest_gate_selection_authority(it validated a non-existenttests/agent_utils). - A nightly run-all
integration-nextlane inci-nightly.ymlrunningpytest tests/integration tests/nextregardless of paths, with red → deduped-P0 escalation (scripts/ci/nightly_escalation.py): onepriority:P0issue per suite key, opened/updated on red and closed on green, fail-closed and token-redacted when the token/API is absent. This retires the deadintegration_tests_nexttier (#4729). - Release gated on a green nightly (
scripts/ci/release_nightly_gate.py, wired inrelease.yml):build-release/publish-pypiare blocked unless the nightly for the exact release SHA is green; fail-closed on a missing, stale, red, or in-progress nightly.
Consequences
- A green nightly now means the full suite ran; a masked per-push red cannot reach a published release, because release gates on that nightly.
- Operator prerequisite:
release.ymldispatches the nightly viaworkflow_dispatch, which the defaultGITHUB_TOKENcannot trigger. A repo secretRELEASE_NIGHTLY_DISPATCH_TOKEN(a PAT or GitHub App token) is required; without it the release gate fails closed and nothing publishes. This is intentional fail-closed behavior.release.yml(Publish Release) is the live release workflow, so the secret must be provisioned before the next tag release — seeRELEASE_CHECKLIST.md. - Per-push selection is still path-filtered (unchanged); this ADR closes the
honesty gap (nothing is untested-by-construction, and release can’t ship a
masked red), not the latency gap. Promoting
tests/integrationto a per-PR lane is deferred to #5037. - The
in-matrix-darkbaseline records remaining nightly-only debt (e.g.diagnostics, imported only bytests/e2e) as an explicit, shrink-only ledger rather than a silent gap.
Alternatives Considered
- Run every module shard on every push. Rejected: prohibitively slow/expensive
for the common case, and it does not address
truly-darkpackages that no test exercises at all (the guards do). - Leave per-push selection as-is and rely on maintainers noticing. Rejected — that is the status quo #5034 documents as failing: reds accumulated invisibly and surfaced only as landing-pass tax on unrelated PRs.
Amendment (2026-10-01) — nightly architectural backstop, sharded battery, CI worker policy (mission ci-runtime-stabilisation, #5510)
This amendment appends to the accepted text above and does not edit it. It corrects one
Consequences claim, records what the mission ci-runtime-stabilisation shipped, and links
the contracts that hold the details:
router two-authority amendment,
battery partition.
Context
This amends the first Consequences bullet, “A green nightly now means the full suite
ran”. That statement did not hold for tests/architectural. Before this amendment the
nightly reached the battery only through the interpreter shard that runs the
fast or unit selection, which is 18.6% of the battery (the SC-003 baseline measured by
the mission). A red architectural gate outside that selection could therefore sit behind a
green nightly. Two open issues name the gap: #4708
(“CI must actually execute the whole test suite”) and
#3265 (no unfiltered main backstop,
folded into this mission). The per-PR battery was also a single slow job that every code PR
waited on.
Decision
- A nightly architectural backstop.
ci-nightly.ymlgains the jobarchitectural-backstop. It runs the full per-PR battery base command (tests/architectural, the base marker expression and the same deselects) with no partition plugin, on Python 3.12, with a 40-minute timeout, so it does not depend on the partition or gate-selection code being correct. It is listed innightly-summary.needs. A red run fails the nightly conclusion, whichscripts/ci/release_nightly_gate.pyalready reads for the release SHA, and it escalates throughscripts/ci/nightly_escalation.pyunder the suite keyarchitectural: one deduplicatedpriority:P0issue per key, triaged on file (type Bug, labelspriority:P0andfrom:ci, milestone, and a native sub-issue link under #5106). - A sharded per-PR battery with an always-on fast gate. The battery is an always-on
architectural-fastjob plusarchitectural-heavyas one job key with a two-leginclude:matrix (--battery-part 1/2and2/2). The registry.github/ci-module-registry.yml(special_tiers.architectural) is the single declaration of the fast roster, the shard count, the worker count and the base selection. The workflows carry literal copies of the worker count, the base marker expression and the deselects, and pin tests (tests/ci/test_xdist_worker_policy.pyand the partition proof) keep those copies equal to the registry. Per-file timings in.github/ci-shard-timings.jsonfeed the shared selectorscripts/ci/shard_select.py. A static partition proof requires fast, leg 1 and leg 2 to be pairwise disjoint and to cover the base selection exactly. - A CI worker policy. Battery legs, the fast gate, the backstop and the Packs corpus job
pass a literal
-n 4and no-q, so xdist printscreated: 4/4 workers.tests/ci/test_xdist_worker_policy.pyfails on-n autoin those jobs. Localmake test-*targets keep-n auto. - A
ci_configpath group. CI-configuration paths (workflows, composite actions,scripts/ci/**,pytest.ini,pyproject.toml,Makefile, the registry and the timings file) select the heavy battery and nothing else. This is amendment A1 of the linked contract. It reverses, for the battery only, the earlier rulings that CI-config changes gate no router job.
Consequences
- The corrected guarantee holds for the architectural battery: a green nightly now means the full battery ran, because the backstop executes the whole per-PR base selection in one plain pytest invocation. Interpreter diversity stays with the interpreter matrix shards.
- The backstop is judged on its own job conclusion. The overall nightly conclusion also reflects unrelated suites, so it cannot show the backstop’s health by itself.
- The per-PR battery stays a non-required check, as ADR 2026-09-23-1 requires. Both the
architectural-fastjob and thearchitectural-heavylegs are in theneedsofrouter-gate, so a red in either turns the requiredrouter gatered. None of these jobs is itself a required context (a path-scoped job is never required, per ADR 2026-09-23-1). - Measurements for the mission’s non-functional requirements are recorded in the mission
evidence file
kitty-specs/ci-runtime-stabilisation-01M3TZH6/evidence/ci-measurements.md.