<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xml:base="https://docs.spec-kitty.ai/adr/3.x/2026-09-26-1-ci-coverage-honesty/">
  <id>https://docs.spec-kitty.ai/adr/3.x/2026-09-26-1-ci-coverage-honesty/</id>
  <title>ADR: a green main/nightly must mean the tests actually ran</title>
  <updated>2026-10-01T00:00:00.000Z</updated>
  <author><name>Spec Kitty</name></author>
  <link rel="alternate" type="text/html" href="https://docs.spec-kitty.ai/adr/3.x/2026-09-26-1-ci-coverage-honesty/"/>
  <summary>CI now enforces coverage honesty — dark-suite enrolment, foreign-coverage and src-reachability guards, a nightly run-all lane, and release gated on a green nightly for that SHA.</summary>
  <content type="html">&lt;h2 id=&quot;context-and-problem-statement&quot;&gt;Context and Problem Statement&lt;/h2&gt;
&lt;p&gt;The per-module CI selector (&lt;code dir=&quot;auto&quot;&gt;scripts/ci/gate_selection.py&lt;/code&gt;, fed by &lt;code dir=&quot;auto&quot;&gt;ci-router.yml&lt;/code&gt;)
keyed which module shards run on a push to &lt;strong&gt;source roots&lt;/strong&gt; — a change under
&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/&amp;#x3C;x&gt;/**&lt;/code&gt; selected the &lt;code dir=&quot;auto&quot;&gt;&amp;#x3C;x&gt;&lt;/code&gt; module — with &lt;strong&gt;no model of which
source a given test exercises&lt;/strong&gt;. Two blind spots followed:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Cross-cutting / unenrolled suites were silently skipped on &lt;code dir=&quot;auto&quot;&gt;main&lt;/code&gt; pushes.&lt;/strong&gt;
Whole test directories (&lt;code dir=&quot;auto&quot;&gt;tests/agent&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tests/specify_cli/live_work&lt;/code&gt;,
&lt;code dir=&quot;auto&quot;&gt;tests/review&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tests/upgrade&lt;/code&gt;, and the &lt;code dir=&quot;auto&quot;&gt;execution_context&lt;/code&gt; family) were
effectively never selected on ordinary &lt;code dir=&quot;auto&quot;&gt;main&lt;/code&gt; pushes, so a regression in code
they cover could ship to a &lt;strong&gt;green &lt;code dir=&quot;auto&quot;&gt;main&lt;/code&gt;&lt;/strong&gt; undetected.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;integration_tests_next&lt;/code&gt; was a dead tier&lt;/strong&gt; — declared in the registry with
roots &lt;code dir=&quot;auto&quot;&gt;tests/integration/**&lt;/code&gt; + &lt;code dir=&quot;auto&quot;&gt;tests/next/**&lt;/code&gt; but invoked by no workflow, so
those suites ran nowhere.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This was root-caused in &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/5034&quot;&gt;#5034&lt;/a&gt;
(epic &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/4437&quot;&gt;#4437&lt;/a&gt;) after it
repeatedly cost maintainer landing passes: PRs #5029 and #5032 each un-masked a
backlog of pre-existing reds the moment their diff touched a path that finally
selected one of those shards. A green check no longer meant the tests had run.&lt;/p&gt;
&lt;h2 id=&quot;decision&quot;&gt;Decision&lt;/h2&gt;
&lt;p&gt;Make “green ⇒ the tests actually ran” an enforced property, in four parts:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Coverage-honesty guards&lt;/strong&gt; (&lt;code dir=&quot;auto&quot;&gt;scripts/ci/coverage_guard_lib.py&lt;/code&gt;, enforced by
&lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_foreign_coverage_guard.py&lt;/code&gt; and
&lt;code dir=&quot;auto&quot;&gt;test_src_reachability_guard.py&lt;/code&gt;) — shrink-only ratchets over a measured
baseline (&lt;code dir=&quot;auto&quot;&gt;.github/ci-foreign-coverage-baseline.json&lt;/code&gt;). &lt;em&gt;foreign-coverage&lt;/em&gt;
requires every registry row’s test dirs to exercise its own &lt;code dir=&quot;auto&quot;&gt;roots:&lt;/code&gt;;
&lt;em&gt;src-reachability&lt;/em&gt; pins a &lt;code dir=&quot;auto&quot;&gt;truly-dark&lt;/code&gt; set (a package imported by no test
anywhere) and an &lt;code dir=&quot;auto&quot;&gt;in-matrix-dark&lt;/code&gt; set (tested only nightly). They fail only
when coverage &lt;strong&gt;regresses&lt;/strong&gt;, never vacuously.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Enrol the dark suites&lt;/strong&gt; into &lt;code dir=&quot;auto&quot;&gt;.github/ci-module-registry.yml&lt;/code&gt; (&lt;code dir=&quot;auto&quot;&gt;agent_utils&lt;/code&gt;,
&lt;code dir=&quot;auto&quot;&gt;live_work&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;config&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;calibration&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tasks_authoring&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;bootstrap&lt;/code&gt;) with
re-measured host shard timings, and fix the phantom-mirror authority bug in
&lt;code dir=&quot;auto&quot;&gt;test_gate_selection_authority&lt;/code&gt; (it validated a non-existent &lt;code dir=&quot;auto&quot;&gt;tests/agent_utils&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A nightly run-all &lt;code dir=&quot;auto&quot;&gt;integration-next&lt;/code&gt; lane&lt;/strong&gt; in &lt;code dir=&quot;auto&quot;&gt;ci-nightly.yml&lt;/code&gt; running
&lt;code dir=&quot;auto&quot;&gt;pytest tests/integration tests/next&lt;/code&gt; regardless of paths, with
&lt;strong&gt;red → deduped-P0 escalation&lt;/strong&gt; (&lt;code dir=&quot;auto&quot;&gt;scripts/ci/nightly_escalation.py&lt;/code&gt;): one
&lt;code dir=&quot;auto&quot;&gt;priority:P0&lt;/code&gt; issue per suite key, opened/updated on red and closed on green,
fail-closed and token-redacted when the token/API is absent. This retires the
dead &lt;code dir=&quot;auto&quot;&gt;integration_tests_next&lt;/code&gt; tier (&lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/4729&quot;&gt;#4729&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Release gated on a green nightly&lt;/strong&gt; (&lt;code dir=&quot;auto&quot;&gt;scripts/ci/release_nightly_gate.py&lt;/code&gt;,
wired in &lt;code dir=&quot;auto&quot;&gt;release.yml&lt;/code&gt;): &lt;code dir=&quot;auto&quot;&gt;build-release&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;publish-pypi&lt;/code&gt; are blocked unless the
nightly for the &lt;strong&gt;exact release SHA&lt;/strong&gt; is green; fail-closed on a missing,
stale, red, or in-progress nightly.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;consequences&quot;&gt;Consequences&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;A green &lt;strong&gt;nightly&lt;/strong&gt; now means the full suite ran; a masked per-push red cannot
reach a published release, because release gates on that nightly.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Operator prerequisite:&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;release.yml&lt;/code&gt; dispatches the nightly via
&lt;code dir=&quot;auto&quot;&gt;workflow_dispatch&lt;/code&gt;, which the default &lt;code dir=&quot;auto&quot;&gt;GITHUB_TOKEN&lt;/code&gt; cannot trigger. A repo
secret &lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;RELEASE_NIGHTLY_DISPATCH_TOKEN&lt;/code&gt;&lt;/strong&gt; (a PAT or GitHub App token) is
required; without it the release gate fails closed and nothing publishes. This
is intentional fail-closed behavior. &lt;code dir=&quot;auto&quot;&gt;release.yml&lt;/code&gt; (&lt;code dir=&quot;auto&quot;&gt;Publish Release&lt;/code&gt;) is the
live release workflow, so the secret must be provisioned before the next tag
release — see &lt;code dir=&quot;auto&quot;&gt;RELEASE_CHECKLIST.md&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Per-push selection is &lt;strong&gt;still path-filtered&lt;/strong&gt; (unchanged); this ADR closes the
&lt;em&gt;honesty&lt;/em&gt; gap (nothing is untested-by-construction, and release can’t ship a
masked red), not the &lt;em&gt;latency&lt;/em&gt; gap. Promoting &lt;code dir=&quot;auto&quot;&gt;tests/integration&lt;/code&gt; to a per-PR
lane is deferred to &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/5037&quot;&gt;#5037&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;The &lt;code dir=&quot;auto&quot;&gt;in-matrix-dark&lt;/code&gt; baseline records remaining nightly-only debt (e.g.
&lt;code dir=&quot;auto&quot;&gt;diagnostics&lt;/code&gt;, imported only by &lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt;) as an explicit, shrink-only ledger
rather than a silent gap.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;alternatives-considered&quot;&gt;Alternatives Considered&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Run every module shard on every push.&lt;/strong&gt; Rejected: prohibitively slow/expensive
for the common case, and it does not address &lt;code dir=&quot;auto&quot;&gt;truly-dark&lt;/code&gt; packages that no test
exercises at all (the guards do).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Leave per-push selection as-is and rely on maintainers noticing.&lt;/strong&gt; Rejected —
that is the status quo #5034 documents as failing: reds accumulated invisibly and
surfaced only as landing-pass tax on unrelated PRs.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;amendment-2026-10-01--nightly-architectural-backstop-sharded-battery-ci-worker-policy-mission-ci-runtime-stabilisation-5510&quot;&gt;Amendment (2026-10-01) — nightly architectural backstop, sharded battery, CI worker policy (mission ci-runtime-stabilisation, #5510)&lt;/h2&gt;
&lt;p&gt;This amendment appends to the accepted text above and does not edit it. It corrects one
Consequences claim, records what the mission &lt;code dir=&quot;auto&quot;&gt;ci-runtime-stabilisation&lt;/code&gt; shipped, and links
the contracts that hold the details:
&lt;a href=&quot;../../../kitty-specs/ci-runtime-stabilisation-01M3TZH6/contracts/router-two-authority-amendment.md&quot;&gt;router two-authority amendment&lt;/a&gt;,
&lt;a href=&quot;../../../kitty-specs/ci-runtime-stabilisation-01M3TZH6/contracts/battery-partition.md&quot;&gt;battery partition&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id=&quot;context&quot;&gt;Context&lt;/h3&gt;
&lt;p&gt;This amends the first Consequences bullet, “A green &lt;strong&gt;nightly&lt;/strong&gt; now means the full suite
ran”. That statement did not hold for &lt;code dir=&quot;auto&quot;&gt;tests/architectural&lt;/code&gt;. Before this amendment the
nightly reached the battery only through the interpreter shard that runs the
&lt;code dir=&quot;auto&quot;&gt;fast or unit&lt;/code&gt; selection, which is &lt;strong&gt;18.6%&lt;/strong&gt; of the battery (the SC-003 baseline measured by
the mission). A red architectural gate outside that selection could therefore sit behind a
green nightly. Two open issues name the gap: &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/4708&quot;&gt;#4708&lt;/a&gt;
(“CI must actually execute the whole test suite”) and
&lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/3265&quot;&gt;#3265&lt;/a&gt; (no unfiltered &lt;code dir=&quot;auto&quot;&gt;main&lt;/code&gt; backstop,
folded into this mission). The per-PR battery was also a single slow job that every code PR
waited on.&lt;/p&gt;
&lt;h3 id=&quot;decision-1&quot;&gt;Decision&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;A nightly architectural backstop.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;ci-nightly.yml&lt;/code&gt; gains the job
&lt;code dir=&quot;auto&quot;&gt;architectural-backstop&lt;/code&gt;. It runs the full per-PR battery base command
(&lt;code dir=&quot;auto&quot;&gt;tests/architectural&lt;/code&gt;, the base marker expression and the same deselects) with no
partition plugin, on Python 3.12, with a 40-minute timeout, so it does not depend on the
partition or gate-selection code being correct. It is listed in &lt;code dir=&quot;auto&quot;&gt;nightly-summary.needs&lt;/code&gt;.
A red run fails the nightly conclusion, which &lt;code dir=&quot;auto&quot;&gt;scripts/ci/release_nightly_gate.py&lt;/code&gt; already
reads for the release SHA, and it escalates through &lt;code dir=&quot;auto&quot;&gt;scripts/ci/nightly_escalation.py&lt;/code&gt;
under the suite key &lt;code dir=&quot;auto&quot;&gt;architectural&lt;/code&gt;: one deduplicated &lt;code dir=&quot;auto&quot;&gt;priority:P0&lt;/code&gt; issue per key,
triaged on file (type Bug, labels &lt;code dir=&quot;auto&quot;&gt;priority:P0&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;from:ci&lt;/code&gt;, milestone, and a native
sub-issue link under &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/5106&quot;&gt;#5106&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A sharded per-PR battery with an always-on fast gate.&lt;/strong&gt; The battery is an always-on
&lt;code dir=&quot;auto&quot;&gt;architectural-fast&lt;/code&gt; job plus &lt;code dir=&quot;auto&quot;&gt;architectural-heavy&lt;/code&gt; as one job key with a two-leg
&lt;code dir=&quot;auto&quot;&gt;include:&lt;/code&gt; matrix (&lt;code dir=&quot;auto&quot;&gt;--battery-part 1/2&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;2/2&lt;/code&gt;). The registry
&lt;code dir=&quot;auto&quot;&gt;.github/ci-module-registry.yml&lt;/code&gt; (&lt;code dir=&quot;auto&quot;&gt;special_tiers.architectural&lt;/code&gt;) is the single declaration
of the fast roster, the shard count, the worker count and the base selection. The workflows
carry literal copies of the worker count, the base marker expression and the deselects, and
pin tests (&lt;code dir=&quot;auto&quot;&gt;tests/ci/test_xdist_worker_policy.py&lt;/code&gt; and the partition proof) keep those
copies equal to the registry. Per-file timings in &lt;code dir=&quot;auto&quot;&gt;.github/ci-shard-timings.json&lt;/code&gt;
feed the shared selector &lt;code dir=&quot;auto&quot;&gt;scripts/ci/shard_select.py&lt;/code&gt;. A static partition proof requires
fast, leg 1 and leg 2 to be pairwise disjoint and to cover the base selection exactly.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A CI worker policy.&lt;/strong&gt; Battery legs, the fast gate, the backstop and the Packs corpus job
pass a literal &lt;code dir=&quot;auto&quot;&gt;-n 4&lt;/code&gt; and no &lt;code dir=&quot;auto&quot;&gt;-q&lt;/code&gt;, so xdist prints &lt;code dir=&quot;auto&quot;&gt;created: 4/4 workers&lt;/code&gt;.
&lt;code dir=&quot;auto&quot;&gt;tests/ci/test_xdist_worker_policy.py&lt;/code&gt; fails on &lt;code dir=&quot;auto&quot;&gt;-n auto&lt;/code&gt; in those jobs. Local
&lt;code dir=&quot;auto&quot;&gt;make test-*&lt;/code&gt; targets keep &lt;code dir=&quot;auto&quot;&gt;-n auto&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A &lt;code dir=&quot;auto&quot;&gt;ci_config&lt;/code&gt; path group.&lt;/strong&gt; CI-configuration paths (workflows, composite actions,
&lt;code dir=&quot;auto&quot;&gt;scripts/ci/**&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;pytest.ini&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;pyproject.toml&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;Makefile&lt;/code&gt;, the registry and the timings
file) select the heavy battery and nothing else. This is amendment A1 of the linked
contract. It reverses, for the battery only, the earlier rulings that CI-config changes
gate no router job.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;consequences-1&quot;&gt;Consequences&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;The corrected guarantee holds for the architectural battery: a green nightly now means the
full battery ran, because the backstop executes the whole per-PR base selection in one
plain pytest invocation. Interpreter diversity stays with the interpreter matrix shards.&lt;/li&gt;
&lt;li&gt;The backstop is judged on its own job conclusion. The overall nightly conclusion also
reflects unrelated suites, so it cannot show the backstop’s health by itself.&lt;/li&gt;
&lt;li&gt;The per-PR battery stays a non-required check, as ADR 2026-09-23-1 requires. Both the
&lt;code dir=&quot;auto&quot;&gt;architectural-fast&lt;/code&gt; job and the &lt;code dir=&quot;auto&quot;&gt;architectural-heavy&lt;/code&gt; legs are in the &lt;code dir=&quot;auto&quot;&gt;needs&lt;/code&gt; of
&lt;code dir=&quot;auto&quot;&gt;router-gate&lt;/code&gt;, so a red in either turns the required &lt;code dir=&quot;auto&quot;&gt;router gate&lt;/code&gt; red. None of these jobs
is itself a required context (a path-scoped job is never required, per ADR 2026-09-23-1).&lt;/li&gt;
&lt;li&gt;Measurements for the mission’s non-functional requirements are recorded in the mission
evidence file &lt;code dir=&quot;auto&quot;&gt;kitty-specs/ci-runtime-stabilisation-01M3TZH6/evidence/ci-measurements.md&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;</content>
</entry>
