Research: CI Coverage Honesty

Phase 0 consolidation. Each decision resolves an unknown surfaced by the spec's requirements. Grounded in the root-cause squad findings (see #5034).

D1 — How does the foreign-coverage guard know a test "exercises" a root? (REVISED post-squad)

1. Per-root coverage with a recorded dark-root baseline: enumerate roots no test in the row's resolved dirs imports (under tests/{module} resolver, the real dark root today is next→specify_cli.runtime). Record that set as a committed baseline; the guard fails only when a NEW dark root appears or a covered root goes dark (regression). It does NOT hard-fail next today — that is honest recorded debt. 2. Recorded per-row baseline ratio with a shrink-only ratchet (mirrors shard_count "MEASURED, never guessed"): the guard fails if a row's own-root ratio drops below its recorded baseline. Improving ratchets it up.

  • Decision: Static import analysis built on specify_cli.ast_analysis.imports primitives (module_of_import_from etc. — the same primitives tests/architectural/test_no_dead_symbols.py uses; there is no turn-key "file imports package P" predicate, the guard builds the walk on top with a full ast.walk so deferred/function-level imports count). For each registry row, resolve its test dirs with module-tests.yml precedence — explicit test_dirs, else tests/<module> — NOT _canonical_test_mirror (using the mirror would check the phantom tests/agent_utils and give a false floor-fail).
  • Two-part gate (REVISED again per F17 — shrink-only baselines, green at baseline):
  • Non-src-root rows are exempt from both gates and recorded as such: ci (roots scripts/ci/, .github/workflows/ — no importable package prefix). Its roots can't map to an import prefix, so the guard skips it explicitly (recorded exemption, not silent).
  • Why this handles the squad's fires: agent (1/78 ratio) and release (3/15) do not have to hit an arbitrary 25% — their low ratio becomes the recorded baseline (honest debt) and cannot regress; src-reachability (D3) separately forces their own source to be reachable by some in-matrix dir. agent's own-root reachability is fixed by FR-004 enrolling tests/specify_cli/agent_utils.
  • Alternatives considered: fixed global threshold (rejected — brittle/flaky, several rows sit at 20–58%, churn drifts them); coverage.py attribution (execution-based, slow, flaky); hand-maintained mapping (drifts).

D2 — Aggregate modules that legitimately host foreign tests

  • Decision: A recorded allow-list keyed by module name (e.g. execution_context, core_misc, unit, specify_cli_runtime — the declared AGGREGATE / TEST-INVENTORY rows whose comments already state they own no single mirror). Each entry carries a one-line dated rationale. The guard skips the ratio gate for allow-listed rows but still applies the ≥1-reachable floor.
  • Rationale: These rows are intentionally cross-cutting (their registry comments say so). Forcing a split would be a false positive. The allow-list is the recorded exception, not a silent pass (spec edge case).
  • Alternatives: no allow-list (false-positives on aggregates, guard gets disabled — worse).

D3 — Src-reachability: which packages are "in scope" and what counts as reachable? (REVISED post-squad)

1. Truly-dark baseline (shrink-only, hard-fail on GROWTH): a package imported by NO test anywhere (in-matrix ∪ out_of_matrix ∪ nightly). F18 measured 23 today — NOT ≈0 — because import-based static analysis structurally cannot see subprocess/CLI-tested modules (specify_cli.__main__, encoding tested via spawned scripts), plus data-packages-with-__init__.py (Cat 1), plus ~17 genuinely test-dark live modules (Cat 3). A hard zero-floor is unsatisfiable, so seed the measured 23 as a committed baseline and fail only when a NEW truly-dark package appears (the strongest ratchet: no new untested-anywhere code lands). WP02 may shrink it by adding tests / deleting dead code. 2. In-matrix-dark baseline (shrink-only): F18 measured 27 (not the ~50 estimate) in-matrix-dark under tests/{module}, many deliberately nightly-only (doctor/proof/state in out_of_matrix) — forcing all per-PR would violate NFR-003. Record as committed baseline; fail only on growth. Remediation SHRINKS it.

  • Decision: Enumerate importable packages by filesystem-walking __init__.py-bearing subpackages under the 6 wheel roots (kernel, glossary, mission_runtime, runtime, specify_cli, charter) *plus loose top-level .py modules directly under those roots. The pyproject.toml [tool.hatch...].packages list is only those 6 top-level entries* — it does NOT enumerate subpackages, so a guard keyed on it alone would catch nothing (it would report specify_cli trivially reachable and miss config/calibration/…). The fs-walk is the correct enumeration; pyproject supplies only the 6 roots to walk under. A package is reachable* if ≥1 test file in an in-matrix dir (a module row's resolved test_dirs, excluding out_of_matrix_test_dirs) imports it (full ast.walk, deferred imports count). Fail closed listing each unreachable package.
  • Exclusions: data-only directories are not packages and are excluded — src/specify_cli/schemas/ contains only wps.schema.json, no __init__.py. It is not import-testable and is already content-tested by tests/specify_cli/core/test_wps_manifest.py; it is dropped from the dark set (the spec's "schemas has zero tests" was wrong — it is content-tested, just not import-tested).
  • live_work is already reachable in-matrix via tests/zeitgeist_client/test_mcp_stdio.py (imports specify_cli.live_work.bindings; tests/zeitgeist_client is claimed by core_misc) — so INV-4 is already green for it. FR-005 (mint the row) is therefore justified by per-PR execution of live_work's own 149-test suite, not by INV-4. (The import is function-level, hence the full ast.walk requirement above.)
  • Two-tier gate (REVISED per F17, then F18 — both tiers are shrink-only baselines, no absolute zero):
  • Data-package exclusion (F18 refinement): exclude any package with no non-__init__ .py module — covers schemas (no __init__.py) AND data packages that have an __init__.py but ship only .yaml/.json (charter.activation.corpus, charter.activation.packs, specify_cli.skills.data).
  • Dead-code finding (F18, follow-up): charter.offering.import_candidates (models.py) has 0 src + 0 test importers — likely dead. Out of this mission's scope (CI honesty, not charter cleanup); recorded as a follow-up, not deleted here.
  • Remediation shrinks the baseline: FR-005 (live_work) + FR-009 (config, calibration, tasks_authoring, diagnostics, bootstrap — their tests exist in out_of_matrix dirs, promote them) remove those 6 from the in-matrix-dark baseline; the guard asserts the baseline shrank by exactly those.
  • Alternatives: hard "100% in-matrix reachable" (rejected — infeasible, contradicts NFR-003 / out_of_matrix design, F17); coverage % (execution-based); pyproject-6-package inventory (catches nothing).

D4 — live_work enrolment (REVISED per F19 — enrol into existing row, not a dedicated row)

  • Decision: Enrol tests/specify_cli/live_work into an EXISTING in-matrix row's test_dirs (core_misc, whose cov_targets include specify_cli) and remove it from out_of_matrix_test_dirs; re-measure that host row's shard timings (capture_shard_timings.py --module core_misc --write) so the added 149 tests' durations are honest (NFR-001).
  • Why not a dedicated live_work row (F19): the WP02 implementer proved that adding a new modules[] row trips a wall of architectural guards — test_module_shard_registry requires a scrub↔registry bijection (needs a live_work group in tests/release/ci_retirement_scrub.json with census evidence) AND test_ci_router_transcription_guards requires a verbatim live_work filter group in ci-router.yml (the auth new-row-via-scrub precedent). That coordinated multi-file change is disproportionate; enrolment into an existing row fully achieves FR-005's per-PR-execution intent. A dedicated shard is a deferred follow-up.
  • #4732 is still satisfied (live_work now runs per-PR); evidence updated to "enrolled for per-PR execution".

D5 — Wiring integration into the nightly (REVISED post-squad)

  • Decision: Add a nightly job in ci-nightly.yml running both tests/integration/ and tests/next/ run-all-regardless (if: always(), exit-into-env pattern like the existing performance/e2e/stress jobs), and retire the dead integration_tests_next special tier in the registry in the same change — the tier already declares BOTH roots but is unwired (#4729); wiring only tests/integration would leave tests/next/ (24 files) dark and leave a dead tier declared inside a CI-honesty mission. Tier reconciliation is mandatory, not optional** (squad GAP-3).
  • Measurement (REVISED): capture_shard_timings.py --module requires a modules[] row and raises KeyError for a non-row dir, so it cannot measure tests/integration. Measure durations with pytest tests/integration tests/next --durations=0 (or add a --test-dirs override to the tool as part of WP-E) and shard only if the monolith overruns the per-shard timeout (NFR-004).
  • Wiring: add the new job to nightly-summary.needs or its red won't surface in the aggregator (squad fit-note 5a).
  • Do NOT add a modules[] row (that would pull it per-PR — out of scope, C-003); this is a dedicated nightly lane.
  • Selection note: run by directory (pytest tests/integration tests/next), NOT by -m integration alone, since 843/1015 are integration-marked but the nightly marker suites don't select integration.
  • Alternatives: per-PR sharded row (rejected — C-003, deferred to #5037); leaving the dead tier (rejected — fresh honesty gap); marker-only (misses the 843).

D6 — Nightly red → P0 escalation mechanism (Decision Moment DM-01M3CZWF)

  • Decision: Both — the nightly job fails loudly (terminal fail-loud step, existing #4212 pattern) AND a helper scripts/ci/nightly_escalation.py opens/updates a priority:P0 GitHub issue via the Actions token, deduped by a stable per-suite key (e.g. a hidden marker <!-- nightly-escalation-key: integration --> in the issue body; search open issues for that key → update the existing one, else create). On green, the helper closes/resolves the issue for that key.
  • Rationale: Operator ruling. Dedup by body-marker search is idempotent (NFR-005). Uses the same SHA-pinned action-token posture as the sonar workflows.
  • Fail-closed: if GITHUB_TOKEN/API is unavailable (fork, outage), the helper logs and exits 0 for the escalation step (degrade to fail-loud only), never crashes the workflow, never echoes the token (C-005). The suite's own red still fails the job.
  • Alternatives: fail-loud only (rejected by operator — reds rely on someone reading logs); new issue per night (rejected — noise, violates NFR-005).

D7 — Release gate on green nightly (FR-008) (REVISED post-squad — GitHub constraints)

  • Decision: A precondition job nightly-gate in release.yml runs scripts/ci/release_nightly_gate.py --tag <release_tag>: it (a) resolves the tag → SHA, (b) checks for an existing ci-nightly.yml run whose head_sha == release_sha that concluded success; if found → pass. If none and --dispatch, it workflow_dispatches ci-nightly.yml by the release TAG ref (workflow_dispatch cannot target a bare 40-hex SHA — ref must be a branch or tag; the tag points at the release SHA so the resulting run's head_sha == release_sha and the poll still matches), passing mode=full, then polls to terminal. Exits non-zero unless the nightly for that SHA concluded success. build-release gains needs: [nightly-gate] (and publish-pypi transitively).
  • Token (critical constraint): a workflow_dispatch issued from Actions with the default GITHUB_TOKEN does NOT start a new run (documented recursion guard), and build-release currently declares only permissions: contents: read. So the dispatch uses a dedicated token — a PAT or GitHub App token in secret RELEASE_NIGHTLY_DISPATCH_TOKEN — and the job adds permissions: actions: write. This is an operator prerequisite (create the secret) and must be called out in the PR body.
  • Fail-closed: missing/red nightly ⇒ no publish. Token/secret absent on a real release ⇒ fail (never silent pass — releases run in-repo). In-flight nightly for the SHA ⇒ wait for terminal result, don't race.
  • Rationale: Makes the honest nightly the release authority (ADR 2026-07-17-1). release-readiness.yml already disclaims owning test evidence.
  • Alternatives: self-dispatch with GITHUB_TOKEN (rejected — silently no-ops, gate would hang/fail); dispatch by bare SHA (rejected — not allowed by GitHub); trust release-readiness.yml (metadata only); manual green-tick input (not enforced).

D8 — Fixing the phantom-mirror authority bug (FR-003)

  • Decision: In test_gate_selection_authority.py, change the tree derivation to default to tests/{module} (which exists: tests/agent) instead of _canonical_test_mirror (which yields the non-existent tests/agent_utils), AND add an assertion that every derived test dir EXISTS on disk (INV-2) so a future phantom mirror fails immediately. Critically (squad GAP-7 / C-002): this keeps the authority test GREEN — the RED must land on the NEW foreign-coverage guard (FR-001), never on the authority gate C-002 protects. FR-003 is a green-preserving fix, not a red-first one.
  • Rationale: The self-referential bug is that both sides used the same mirror function; defaulting to the real tests/{module} + an on-disk existence assertion breaks the self-reference without reddening the authority gate.

Supply-chain note

No new third-party runtime dependencies are added. New scripts use the stdlib + already-vendored PyYAML + the GitHub Actions token via gh/REST (already used across scripts/ci/). Actions are SHA-pinned per DIR-051 (as sonar.yml vets). No preinstall/lifecycle-script surface. Adversarial-evidence pass: no security-impacting dependency decision — nothing contested; recorded as N/A per contracts/adversarial-evidence-contract.md.

Adversarial evidence ledger (brownfield point-cut, post-plan)

Three-lens brownfield squad (fit / correctness / scope+regression). Every contested finding + disposition (no silent drops):

#FindingDispositionWhere folded
F1Foreign-coverage: WP-B (promote tests/specify_cli/agent_utils) leaves agent RED at ~2.5% — fixed 25% won't green itchangedD1: per-root floor + recorded-baseline ratchet (no fixed threshold); agent's low ratio becomes honest baseline; src-reachability forces agent_utils reachable
F2release (20%) and ci (non-src roots) fire the ratio gate with no remediation WPchangedD1: baseline ratchet records release's ratio; ci exempt (non-src roots, recorded)
F3src-reachability keyed on pyproject 6-package list catches nothingchangedD3: fs-walk __init__.py subpackages + loose modules, not pyproject list
F4schemas is a data dir (no __init__.py), not importable; already content-testedacceptedD3 + spec FR-009/SC-002: dropped from dark set
F5live_work already reachable via tests/zeitgeist_client (function-level import)acceptedD3: FR-005 rejustified to per-PR execution; full ast.walk pinned
F6Foreign-coverage floor should be per-root for multi-root rows (next etc.)acceptedD1: per-root floor
F7Resolver must use module-tests.yml precedence, not _canonical_test_mirroracceptedD1 + guards-contract
F8capture_shard_timings --module KeyErrors on non-row integrationchangedD5: measure via pytest --durations; optional --test-dirs
F9mode="main" doesn't exist in gate_selection (only full/pr)acceptedquickstart examples fixed to mode="pr"
F10Release workflow_dispatch can't target bare SHA; GITHUB_TOKEN won't trigger nested run; build-release has contents:read onlychangedD7: dispatch by tag ref + dedicated actions:write token secret; operator prerequisite
F11GAP-2 (HIGH): RED-first guards join the always-on architectural battery — merging WP-A alone reds every repo PRchangedSee D9 below: atomic single-PR landing; RED-first shown in commit history only
F12GAP-3: wiring only tests/integration drops tests/next; dead integration_tests_next tier left declaredchangedD5: enroll both + retire tier (mandatory)
F13GAP-5: guard ≤5 s budget unproven for a full-corpus import scanacceptedD1: scope the scan to resolved test_dirs + src walk; cache like test_no_dead_symbols if needed (impl note)
F14GAP-6: 26 loose top-level specify_cli/*.py (incl. agent_tasks_ports.py, #5034) not honestly enrolled; covered only by fail-closed catch-alldeferred_with_rationaleD3 extends reachability to loose modules (cheap wins, e.g. agent_tasks_ports is reachable via tests/review); any residual not resolvable by enrolment is an accepted residual under the catch-all, follow-up noted
F15data-model.md said agent gains tests/agent_utils (phantom)accepteddata-model corrected to tests/specify_cli/agent_utils
F16Nightly integration job must be added to nightly-summary.needsacceptedci-workflow-contract
F17WP01 implementer (impl point-cut, HIGH): under the mandated tests/{module} resolver the expected-RED sets are wrong — agent per-root floor already met (tests/agent/test_agent_utils_status.py:13), real dark root is next→specify_cli.runtime, and ~50 pkgs in-matrix-dark (not 5, many deliberately nightly-only). Hard red-first guards would stay RED after the mission and red every repo PR.changedD9 (green-at-baseline supersedes red-first), D1 (dark-root baseline), D3, spec SC-002 + US1/US2 + FR-004, guards-contract, WP01/WP02 prompts
F18WP01 implementer (impl point-cut, 2nd pass, HIGH): truly-dark = 23, not ≈0 (import analysis can't see subprocess/CLI-tested modules; data-pkgs-with-__init__; ~17 real test-dark). A hard zero-floor is unsatisfiable → conflicts with green-at-baseline. In-matrix-dark measured 27 (not ~50). Also found likely dead code charter.offering.import_candidates.changedD3 (truly-dark becomes shrink-only baseline; data-pkg exclusion refined to "no non-__init__ .py"), guards-contract, spec SC-002 + US2, data-model INV-4, WP01 T003. Dead-code = follow-up.
F19WP02 implementer (impl point-cut, HIGH): WP02's registry-only ownership can't (a) mint a dedicated live_work row — needs scrub-JSON + ci-router edits guarded by bijection/transcription; (b) enrol diagnostics — imported only by path-routed tests/e2e, no in-matrix dir; (c) retire the integration_tests_next tier — test_special_tiers_encode_heavy_pole_deserialization hard-requires it, no in-mission restorer. Clean shrink-by-5 proven green.changedD4 (live_work → enrol into existing row + re-measure host, not dedicated); spec FR-005/FR-009/SC-002 (shrink 5, diagnostics=recorded debt e2e-covered); T011 moved WP02→WP03 (+repoint the guard, +own test_module_shard_registry.py); tasks/WP02/WP03
—Supply-chain adversarial passN/Ano security-impacting dependency decision

D9 — Green-at-baseline landing (REVISED — supersedes RED-first; resolves squad GAP-2 + implementer finding F17)

  • Decision: The guards land GREEN at a recorded baseline, NOT red-first. The WP01 implementer proved (F17) that under the mandated tests/{module} resolver the expected-RED sets are wrong — agent's per-root floor is already met (tests/agent/test_agent_utils_status.py:13 imports specify_cli.agent_utils.status), next's specify_cli.runtime root is the real dark root, and ~50 packages are in-matrix-dark (not 5), many deliberately nightly-only (tests/doctor/proof/state in out_of_matrix). A hard "zero-dark / red-first" guard would stay RED after the full mission and — since tests/architectural/ is always_on — red every repo PR.
  • Consequence: both guards are shrink-only ratchets over a committed measured baseline (the philosophy already chosen for INV-1b). They pass at baseline (recording honest current debt) and fail only on a regression (a NEW dark package/root). Remediation (FR-004/005/009) shrinks the baseline; the guard asserts the shrink. This dissolves GAP-2 entirely — no red guard ever reaches main, so no atomic-landing constraint is needed (though landing the mission in one PR is still tidy). FR-003 authority fix stays green-preserving (D8).
  • Honesty floor preserved: the src-reachability guard additionally enforces a hard truly-dark floor — a package tested nowhere (in-matrix OR out_of_matrix OR nightly) fails immediately (fail-closed on the real honesty violation). schemas is excluded (data-only). This is the "green main means the tests exist and ran somewhere" guarantee; the in-matrix-dark baseline is the "no new masking" ratchet on top.