Mission Specification: Reconcile-flake family: transient same-SHA CI reconcile failures
Mission Branch: fix/reconcile-flake-family-4882 Created: 2026-09-22 Status: Draft Input: Epic #4882 ("Reconcile-flake family: transient same-SHA CI reconcile failures across three job shapes"), owning three open child issues: #4652 (report-main), #4675 (CI Aggregate / collect), #4878 (per-PR report (<pr>)).
Clarifications / Decision Record
These are binding operator decisions, made before this spec was written. They constrain everything below; do not re-derive or second-guess them during plan/tasks/implementation. A future reader (including the R1–R6 review squad auditing this spec) should be able to see exactly what was decided and why from this section alone.
Decision 1 — Scope is the whole epic, one root-cause fix, not three point-fixes
The operator explicitly chose epic scope: fix the shared defect class ("read racy CI evidence, treat any instability as fatal, no bounded retry") across all three job shapes in one mission, rather than a narrower point-fix scoped to #4878 alone. All three child issues — #4652 (report-main job, scripts/ci/fleet_main.py::report()), #4675 (CI Aggregate / collect job, scripts/ci/reconcile_shards.py::main()), and #4878 (per-PR report (<pr>) job, scripts/ci/fleet_verdict.py::report()) — MUST be covered by this mission's functional requirements. Closing only one or two of the three child issues does not satisfy this mission.
Decision 2 — Verification strategy: unit tests over mocked racy sequences, plus a post-merge watch
The operator selected deterministic unit tests over mocked racy/changing API or artefact-presence response sequences as the pre-merge verification mechanism, exercising the retry/backoff logic directly (including both safety invariants in "Requirements" below, under simulated instability), plus a post-merge observation window over real main-head CI cycles as the actual confirmation that the flake family has stopped recurring.
Two alternatives were considered and explicitly REJECTED by the operator. Neither may appear anywhere in this spec, or in downstream plan/tasks/implementation artifacts, as a proposed mechanism — not even as a "nice to have" or "future work" note:
- (a) Adding a manually-dispatchable rehearsal/replay path to exercise the fix.
- (b) Splitting this into a separate observability-first mission.
First-class limitation (not a footnote): this mission's own fix cannot be exercised by its own pull request. All three affected workflows fire on workflow_run events. A workflow_run-triggered workflow run always executes the copy of the workflow file, and any scripts it references, that live on the repository's default branch (main) at trigger time — never the copy carried by the triggering PR's own branch. This is a structural property of the GitHub Actions workflow_run trigger, not a gap in this mission's test plan. Consequences:
- A green PR for this mission is not evidence that the race is closed — the PR's own CI never runs the PR's own version of
fleet_verdict.py,fleet_main.py, orreconcile_shards.pythrough the realworkflow_runtrigger path; it can only run them as directly-invoked unit tests. - The only evidence obtainable before merge is the unit-test coverage from Decision 2 above.
- Confirmation that the real race is closed is only obtainable after merge, by watching real
main-head CI cycles run the merged code through its actualworkflow_runtriggers. This is genuine post-merge observation, not a pre-merge mission-acceptance gate — Success Criteria below states it as an observation window, not something this mission's own PR must demonstrate to be considered done.
User Scenarios & Testing (mandatory)
User Story 1 - Stop training reviewers to dismiss tip-red as noise (Priority: P1)
As a maintainer or review agent watching main-head CI, I want the three reconcile steps (report-main, CI Aggregate collect/dedup, per-PR report (<pr>)) not to flap red-then-green at an unchanged SHA, so that a red CI Fleet Verdict or CI Aggregate check-run keeps meaning "something is actually wrong" instead of training everyone to dismiss it as "just the flake" — the exact habit that would let a genuine red slip through unnoticed.
Why this priority: This is the epic's entire stated harm (signal erosion, not merge-gating), and the epic explicitly says recurrence is worsening ("the 'first attempt fails, retry passes' pattern no longer reliably holds" — #4675's most recent comment), so the erosion is compounding, not settling on its own.
Independent Test: Can be tested independently of any specific job shape by running the new unit-test suite(s) added for FR-002/FR-003/FR-004 (the fleet_verdict.py/fleet_main.py double-snapshot retry) and FR-005 (the new ci-aggregate.yml artefact-visibility polling step/script) below and confirming they exercise racy/changing mocked evidence sequences and assert the retry/backoff logic recovers without a false report, plus the unit tests confirming reconcile_shards.py::main()'s unchanged single-pass fail-closed behavior (FR-006, SC-001), and — per Decision 2 — by observing real main-head cycles post-merge for the absence of same-SHA reconcile flaps across all three job shapes.
Acceptance Scenarios:
1. Given a mocked evidence source that returns an unstable/changing result on the first read and a stable result on a subsequent read within the retry budget, When any of the three retry-bearing reconcile surfaces (fleet_verdict.py::report(), fleet_main.py::report(), or the new ci-aggregate.yml artefact-visibility polling step/script) runs against it, Then the call/step succeeds without raising and without posting/reporting/downloading based on the unstable first read. 2. Given reconcile_shards.py::main() running unmodified against whatever the (possibly retried) download step actually produced — a single, already-static local directory with no network/API access — When main() performs its one-pass completeness check, Then it succeeds (exit 0) if every selected/must_be_fresh shard is present in that directory, and fails closed (non-zero exit, ::error:: annotation) if one is genuinely absent; in neither case does main() itself retry, poll, or re-read the directory a second time. 3. Given a real post-merge main-head CI cycle, When the three reconcile steps run through their actual workflow_run triggers, Then no same-SHA fail-then-pass sequence is observed for any of the three job shapes across the bounded post-merge watch window defined in Success Criteria.
User Story 2 - fleet_verdict.py / fleet_main.py pair retries the whole double-snapshot as a unit, never posts stale evidence (Priority: P1)
As a maintainer relying on the CI Fleet Verdict comment/incident to be truthful, I want the shared double-snapshot idiom in fleet_verdict.py::report() (per-PR, #4878) and its hand-duplicated counterpart in fleet_main.py::report() (main-push P0 intake, #4652) to retry the entire snapshot-compare-publish sequence a bounded number of times on instability, and to skip silently (no comment, no incident, exit 0) rather than raise, when the retry budget is exhausted and a later workflow_run event will naturally re-fire this same reconcile — so that the comment/incident stream is never built from stale or mid-flight evidence, and the current fail-fast-and-defer design ("later event will reconcile") keeps its guarantee while losing its externally-visible flake.
Why this priority: Covers two of the three child issues (#4878, #4652) and the fan-in workflow (ci-fleet-verdict.yml) that fires repeatedly per head (once per completing producer workflow: CI Router, CI Modules, Packs, CI Quality, ci-windows, Release Readiness Check, Check Shared Package Drift, CI Aggregate — per ci-fleet-verdict.yml's workflow_run.workflows list), which is what makes "skip and let a later event redo it" a safe strategy here.
Independent Test: Can be tested independently by unit-testing fleet_verdict.py::report() and fleet_main.py::report() in isolation, mocking snapshot()/GitHub calls to return a changing result across the two internal snapshot calls inside the retry loop, and asserting: (a) the function retries the full snapshot pair, not just the second half; (b) on eventual stability within budget it publishes correctly; (c) on exhausted budget it returns without raising and without an API write (comment/incident creation), rather than raising ValueError as today.
Acceptance Scenarios:
1. Given snapshot() returns evidence A on read 1 and evidence B (different) on the re-check read, and evidence stabilizes on a subsequent retry within the bounded budget, When fleet_verdict.py::report() or fleet_main.py::report() runs, Then it posts the comment/incident using the stabilized evidence, not evidence A. 2. Given snapshot() never stabilizes across the full bounded retry budget, When either report() function runs, Then it returns (exit 0 for the invoking main()) without posting a comment or opening/updating an incident, and without raising — the observable is: no api.request(...) write call is made, and the process exit code is 0. 3. Given the same exhausted-budget scenario, When the reconcile step's log is inspected, Then it prints a diagnostic line to stdout (a plain print(...) call, matching the existing print(...)/::error::/::warning:: annotation conventions already used by fleet_verdict.py/fleet_main.py/reconcile_shards.py — not raised as an exception, not silent) containing, at minimum, the subject identifier (PR number or main-head SHA), the word "deferred" or "skipped", and the reason "evidence did not stabilize within retry budget" — this is a logging/diagnostic requirement, not a change to the exit code or return value. 4. Given two concurrent invocations of fleet_verdict.py::report() (or fleet_main.py::report()) for two different subjects — e.g., PR #4878 and PR #4652, or a PR vs. a main-head push — each under simulated instability, When both run concurrently in the test, Then neither invocation reads or writes any shared object, global variable, or filesystem path belonging to the other: each invocation's retry state (attempt counters, backoff timers, snapshot buffers) is constructed fresh from its own arguments and neither invocation's outcome depends on the other's timing or evidence (this is the FR-009 no-cross-run-coupling property).
User Story 3 - ci-aggregate.yml retries its own-run artefact-visibility check before reconcile_shards.py fails closed (Priority: P1)
As a maintainer relying on the CI Aggregate coverage gate to be complete, I want ci-aggregate.yml to gain a new bounded polling step — placed ahead of its existing actions/download-artifact steps — that re-checks GitHub Actions artefact visibility for the triggering workflow_run's expected shard names a bounded number of times with backoff before falling through to the existing download-and-reconcile steps, because ci-aggregate.yml fires on a single, non-repeating workflow_run event per CI Modules completion — there is no later event to defer to the way the fleet-verdict pair has. reconcile_shards.py::main() itself is unchanged by this requirement: it remains a pure, already-tested function over an already-downloaded local directory (per its own module docstring — extracted "so the completeness decision is unit-testable") with no network/API access, so a retry loop re-reading that same static directory could never observe a different answer; the retryable read belongs to the workflow step that populates the directory, not to main(). reconcile_shards.py::main() continues to fail loudly (non-zero exit, ::error:: annotation) if a selected (must_be_fresh) shard is genuinely still absent from whatever the (possibly retried) download actually produced, so the must_be_fresh fail-closed completeness floor — documented under "Stale-artefact fallback" in contracts/artefact-naming.md and in reconcile_shards.py's own ::error:: annotation — is never weakened.
Why this priority: Covers the third child issue (#4675) and is structurally distinct from User Story 2 — reconcile_shards.py::main() does a single-pass completeness check today (no double-snapshot) and has no network access to retry against, and the workflow that invokes it has no natural "later event" to skip-and-defer to, so this surface needs a real bounded wait in the workflow, ahead of the download step — not a skip, and not a retry loop inside main().
Independent Test: Can be tested independently by unit-testing the new artefact-visibility polling step/script (e.g. scripts/ci/wait_for_artifacts.py, mirroring the mocked GitHub/GitHubCLI boundary already used by scripts/ci/fleet_verdict.py) with a mocked GitHub Actions artifacts-API client that reports a selected/must_be_fresh shard's artefact as absent on the first N-1 polls and present by the Nth, within budget, and separately with a client that never reports it present within budget — asserting the polling step's own bounded termination in both cases — plus unit-testing reconcile_shards.py::main() unchanged, confirming its existing fail-closed exit and ::error:: annotation still fire whenever what was actually downloaded lacks a selected shard, regardless of how the download step arrived at that state.
Acceptance Scenarios:
1. Given a selected (must_be_fresh) shard's artefact that is absent from the GitHub Actions API on the first poll but present on a retry within the bounded budget, When the new artefact-visibility polling step runs ahead of actions/download-artifact, and actions/download-artifact and reconcile_shards.py::main() then run as today, Then the now-present shard is downloaded and reconcile_shards.py::main() completes successfully (exit 0) using it, with no ::error:: annotation. 2. Given a selected (must_be_fresh) shard's artefact that remains absent across the entire bounded polling budget, When the polling step exhausts its budget and the existing download-and-reconcile steps then run against whatever is actually available, Then reconcile_shards.py::main() still exits non-zero and still prints the ::error::ci-aggregate: ... refusing to silently treat this run as complete ... annotation (or an equivalent that preserves the same fail-closed semantics) — this is the must_be_fresh fail-closed completeness floor (documented under "Stale-artefact fallback" in contracts/artefact-naming.md and in reconcile_shards.py's own ::error:: annotation), and it MUST NOT regress to a 0/success exit under any retry-exhaustion path.
Edge Cases
- Retry budget exhausted, genuine terminal instability (not a transient race): for the
fleet_verdict.py/fleet_main.pysurface, this means skip silently and exit 0, trusting a laterworkflow_runevent to reconcile (per User Story 2) — this is a deliberate, already-designed behavior the fix preserves, not a new failure mode. Forreconcile_shards.py, this means fail loudly (non-zero exit,::error::annotation) per User Story 3 / invariant (b) — there is no later event to defer to, so silent success would be a real defect (a violation of themust_be_freshfail-closed completeness floor documented under "Stale-artefact fallback" incontracts/artefact-naming.md), and silent skip is not an option this surface has. - Concurrent missions' CI runs sharing the same reconcile code: this repository's own CI reporting (the reconcile scripts under fix) is itself exercised by every concurrently-running mission's PR and by
main-head pushes while this mission is in flight. The retry/backoff behavior MUST NOT introduce cross-run state, shared mutable files, or any coupling between one PR's/one head's reconcile invocation and another's — each invocation ofreport()(fleet_verdict.py/fleet_main.py) or of the newci-aggregate.ymlartefact-visibility polling step retries only its own evidence read for its own subject (PR number,mainhead, or aggregate run id), never blocking on or being blocked by another concurrently-running reconcile. (reconcile_shards.py::main()is out of scope for this isolation requirement — it does not retry at all post-fix, so it has no retry state to isolate.) A fix that adds, e.g., a repo-wide lock file or a shared retry-state artifact would violate this. - A genuine GitHub API outage (not mere raciness) during the retry window: the retry/backoff logic MUST still terminate within its bounded budget and then follow the same terminal-failure path each surface already has (skip-and-defer for the fleet-verdict pair; fail loudly for
reconcile_shards.py) — it must NOT retry forever, and it must NOT be indistinguishable from "the fix is broken" (i.e., it should not spin past its bounded budget just because the underlying cause is an outage rather than a race; the bound is on attempts/time, not on diagnosing which cause is in play). - A comment/incident IS successfully posted, and then a late-arriving, now-current
workflow_runevent fires the same reconcile again with newer evidence: this is the existing, already-relied-upon "later event will reconcile" mechanism for the fleet-verdict pair (the retry/backoff logic in FR-003/FR-004 must not interfere with it — it is a distinct, separate reconciliation pathway, not something this mission's bounded in-process retry replaces). - A shard is legitimately, permanently missing (e.g., a selected module's job genuinely failed to produce output, not a timing race): the new
ci-aggregate.ymlpolling step's bounded retry must not convert this into a false "eventually present" success — the retry only covers the visibility-timing window beforeactions/download-artifactruns, and once the bounded budget is spent without the artefact appearing,reconcile_shards.py::main()'s guard MUST still fail exactly as it does today (this is the samemust_be_freshfail-closed completeness floor as above, stated from the opposite direction: retrying does not mean "retry is now doing the guard's job").
Requirements (mandatory)
Functional Requirements
| ID | Title | User Story | Priority | Status |
|---|---|---|---|---|
| FR-001 | Cover all three job shapes | As a maintainer, I want #4652 (report-main), #4675 (CI Aggregate collect/dedup), and #4878 (per-PR report (<pr>)) all addressed by this mission's fix, so that no child issue of epic #4882 remains open for an uncovered job shape. | High | Open |
| FR-002 | Bounded retry on fleet_verdict.py::report()'s double-snapshot | As a maintainer, I want fleet_verdict.py::report() to retry its full snapshot→compare→(re-snapshot→compare)→publish sequence up to a bounded number of attempts when the two snapshots disagree, instead of raising ValueError on the first disagreement, so that a transient mid-flight change no longer surfaces as a failed job when the evidence would have stabilized within a short retry window. | High | Open |
| FR-003 | Bounded retry on fleet_main.py::report()'s double-snapshot | As a maintainer, I want fleet_main.py::report() (a separately hand-written, not-shared duplicate of the same double-snapshot idiom) to gain the equivalent bounded retry as FR-002, so that #4652's report-main step stops flapping for the same reason #4878's report (<pr>) step does. | High | Open |
| FR-004 | Skip-and-defer on exhausted retry budget, fleet-verdict pair only | As a maintainer, I want both fleet_verdict.py::report() and fleet_main.py::report() to, when the retry budget in FR-002/FR-003 is exhausted without stabilizing, return without posting a comment/incident and without raising (exit 0), relying on ci-fleet-verdict.yml's repeated workflow_run firings (once per producer workflow completion) to reconcile on a later event, so that a genuinely-unstable window degrades to "silently deferred, retried on the next natural trigger" instead of a red job. | High | Open |
| FR-005 | Bounded artefact-visibility retry ahead of reconcile_shards.py's completeness guard | As a maintainer, I want ci-aggregate.yml to gain a new bounded polling step, placed ahead of its existing actions/download-artifact steps, that re-attempts a GitHub Actions artefact-visibility check for the expected shard names a bounded number of times with backoff before falling through to the existing download-and-reconcile steps, because ci-aggregate.yml fires once per non-repeating workflow_run (from CI Modules) with no later event to defer to, so the fix here must resolve the timing race in the CI pipeline, ahead of reconcile_shards.py::main() — not by adding retry logic inside main() itself. Ownership of the retryable network call: this new polling step/script owns it. reconcile_shards.py::main() owns no network/API call and is not modified by this requirement — it remains a pure function over an already-downloaded local directory (deliberately extracted to stay unit-testable; see its own module docstring), which has no capability to observe a different answer between two calls in the same job. Implementation shape: left to the plan phase, per the same pattern FR-008 uses for the fleet-verdict shared-helper-vs-duplication decision. What is binding here is the outcome, not a specific module: the polling step's implementation MUST be pytest-mockable and stdlib-only (NFR-002), with unit tests added in tests/ci/ against its own GitHub Actions artifacts-API client (NFR-003(3)). An in-repo, importable Python script/module under scripts/ci/ (illustratively, e.g. scripts/ci/wait_for_artifacts.py, invoked from a ci-aggregate.yml run: step) is the stated preference for meeting that outcome, because it is the shape most straightforwardly pytest-mockable and easiest to keep stdlib-only; inline YAML/bash polling logic and a third-party GitHub Marketplace retry Action are disfavored for that same reason but are not foreclosed outright — the plan phase weighs them against the NFR-002/NFR-003 outcome and makes the final call, exactly as FR-008 leaves its analogous decision open. This polling step/script is a fourth entity this mission is expected to add (see Key Entities). | High | Open |
| FR-006 | Fail-closed floor preserved on reconcile_shards.py artefact-visibility retry exhaustion | As a maintainer, I want reconcile_shards.py::main() to still exit non-zero and still print its ::error::ci-aggregate: ... refusing to silently treat this run as complete ... annotation (citing the must_be_fresh fail-closed completeness floor documented under "Stale-artefact fallback" in contracts/artefact-naming.md and in reconcile_shards.py's own comment) when the FR-005 polling budget is exhausted and a selected shard is genuinely still absent from what actions/download-artifact fetched, so that the fix never regresses the repo's dominant failure mode (silent success) by converting a real missing-shard defect into a quiet pass. | High | Open |
| FR-007 | Never post from stale evidence — invariant (a) | As a maintainer, I want the FR-002/FR-003 retry to always retry the entire snapshot pair as one unit and never catch the "evidence changed between snapshots" condition to fall back to posting the first (stale) snapshot's evidence, so that classify()'s documented guarantee ("Never turn absent, cancelled, skipped or incomplete evidence into green") is not silently reintroduced as a false-report risk by the retry logic itself. | High | Open |
| FR-008 | Consider (not silently re-duplicate) the fleet_verdict.py/fleet_main.py shared-logic gap | As a maintainer, I want the plan phase to explicitly weigh introducing a shared retry/defer helper used by both fleet_verdict.py::report() and fleet_main.py::report(), versus hand-patching FR-002 and FR-003 as two parallel, independently-maintained copies (the same pattern that let the current double-snapshot logic drift into two hand-written copies in the first place), so that this fix does not repeat the duplication it is itself evidence of. | Medium | Open |
| FR-009 | No cross-run/cross-mission coupling | As a maintainer running multiple concurrent missions whose PRs and main-head pushes all exercise this same reconcile code, I want each retry/backoff invocation scoped only to its own subject (one PR number, or the current main head, or one aggregate run id) with no shared mutable state, lock file, or coupling to another concurrently-running invocation, so that this mission's fix does not introduce a new failure mode where one mission's CI reporting blocks or corrupts another's. | High | Open |
Non-Functional Requirements
| ID | Title | Requirement | Category | Priority | Status |
|---|---|---|---|---|---|
| NFR-001 | Retry MUST be bounded | Every retry/backoff loop introduced by this mission (FR-002, FR-003, FR-005) MUST have a fixed, finite maximum number of attempts and/or a fixed maximum total wait time — never an unbounded or infinite retry. Exact attempt counts and backoff timing are a plan-phase decision, not prescribed here, but "bounded" itself is a hard requirement, testable by a unit test that asserts the loop terminates (with the documented terminal behavior from FR-004/FR-006) when the mocked evidence source never stabilizes. | Reliability | High | Open |
| NFR-002 | No new runtime dependency | The retry/backoff implementation MUST use only the Python standard library (e.g., time.sleep plus a bounded loop, or an equivalent stdlib-only construct) — it MUST NOT add a new third-party dependency (e.g., tenacity, backoff). This keeps uv.lock freshness and any associated dependency-audit gate entirely out of scope for this mission's CI verification. | Technical | High | Open |
| NFR-003 | Unit-test coverage of retry/backoff logic under simulated raciness | Each of the following surfaces MUST have unit tests added in tests/ci/ that mock the relevant evidence source across the retry window, and assert both the recovery path (stabilizes within budget) and the terminal path (budget exhausted) documented in FR-004/FR-006/FR-007: (1) FR-002/FR-004 combined for fleet_verdict.py::report(), mocking its snapshot()/GitHub API calls; (2) FR-003/FR-004 combined for fleet_main.py::report(), mocking its equivalent snapshot()/GitHub API calls; (3) FR-005/FR-006 for the new ci-aggregate.yml artefact-visibility polling step/script, mocking its own GitHub Actions artifacts-API client (mirroring the GitHub/GitHubCLI boundary already mocked for fleet_verdict.py) — not reconcile_shards.py::main()'s already-static local-directory read, which this mission does not change; and (4) FR-009 — unit tests assert that two concurrent invocations of report()/main() for two different subjects (distinct PR numbers, or a PR vs. a main-head push) do not read or write any shared object, global, or filesystem path, each invocation's retry state constructed fresh from its own arguments. Each new retry unit test MUST first be run against the unmodified (pre-fix) surface and confirmed to fail the way the current code does today (e.g., the existing ValueError from fleet_verdict.py/fleet_main.py, or a non-zero exit with no retry, on the very first unstable/absent read) before the retry implementation is added, per Charter Standing Order #4 (red-first through the pre-existing entry point). This is the pre-merge verification mechanism per Decision 2; no coverage-percentage floor applies (see Constraints), but the tests themselves are required regardless. | Reliability | High | Open |
Constraints
| ID | Title | Constraint | Category | Priority | Status |
|---|---|---|---|---|---|
| C-001 | workflow_run always runs the default-branch copy | All three affected workflows (ci-fleet-verdict.yml, ci-aggregate.yml) trigger on workflow_run, which always executes the main-branch copy of the workflow file and any scripts (scripts/ci/*.py) it references — never the triggering PR's own copy. This mission's own PR therefore cannot exercise the fix through its real trigger path; pre-merge verification is limited to the unit tests in NFR-003 (restated from the Clarifications / Decision Record, since it directly bounds what "done" can mean for this mission's own PR). | Technical | High | Open |
| C-002 | Fail-closed floor for reconcile_shards.py is inviolable | No implementation of FR-005 may weaken or remove the must_be_fresh fail-closed completeness floor ("never silently treat as complete", documented under "Stale-artefact fallback" in contracts/artefact-naming.md and in reconcile_shards.py's own ::error:: annotation) in reconcile_shards.py for a genuinely, permanently absent selected shard. Widening the window before giving up (via the FR-005 bounded polling step) is in scope; removing or softening reconcile_shards.py::main()'s eventual failure is not. | Technical | High | Open |
| C-003 | Never post from a stale snapshot | No implementation of FR-002/FR-003 may catch the current ValueError("... changed before publication; later event will reconcile") condition and fall through to publishing using the earlier (stale) snapshot's evidence instead of retrying the whole pair. This is invariant (a) restated as a hard constraint on the implementation shape, not just an acceptance criterion. | Technical | High | Open |
| C-004 | Stdlib-only retry implementation | Restated from NFR-002 as a constraint on implementation choice: no new third-party dependency may be introduced for retry/backoff. | Technical | Medium | Open |
| C-005 | ruff check . and ruff format --check . are both hard-enforced | Both run with no continue-on-error in ci-quality.yml's lint job on every PR touching this code. Run make format-check before every commit. mypy does not run in CI at all and is not a gate here. SonarCloud (sonar-pr in ci-aggregate.yml) is continue-on-error and excluded from the terminal aggregate-gate — reported, not required. | Technical | Medium | Open |
| C-006 | Reflexivity: this mission's own machinery is what's being fixed | The three scripts under fix are themselves what report CI status for every concurrently-running spec-kitty mission, including this one. The fix must not regress reconcile reporting for any mission whose PR or main-head push is in flight while this mission is being developed or after it merges (see also FR-009 and the concurrency edge case). | Technical | High | Open |
| C-007 | No ledger entry exists for this behavior | The mission orchestration workspace's SPEC-KITTY-LEDGER.md (maintained outside this checkout, not part of the spec-kitty repository) was grepped for "reconcile", "fleet.verdict", "fleet_main", and "reconcile_shards" with no relevant hits — this defect family has no prior ledger record. Noted here so a reviewer does not need to re-grep it. | Documentation | Low | Open |
| C-008 | Diff-cover coverage floor does not apply, but tests are still required | Changes confined to scripts/ci/, .github/workflows/{ci-fleet-verdict,ci-aggregate}.yml, and tests/ci/ route to the ci-modules.yml ci shard, whose cov_targets (.github/ci-module-registry.yml) are kernel/specify_cli.core — scripts/ci/** is not --cov-instrumented, so the diff-cover ≥90% gate structurally does not apply to this mission's diff. This does NOT excuse omitting tests: the repo's standing order ("every new branch/helper needs tests in the same PR") still applies and is enforced by review, not by the coverage gate. | Technical | Medium | Open |
Key Entities
Not applicable in the conventional sense — this is a CI-infrastructure defect fix with no new domain/data entities. Four script-level "entities" are relevant:
1. scripts/ci/fleet_verdict.py (per-PR report()) — unchanged in shape; only its retry behavior around its existing double-snapshot read changes (FR-002/FR-004). 2. scripts/ci/fleet_main.py (main-push report()) — unchanged in shape; only its retry behavior around its existing double-snapshot read changes (FR-003/FR-004). 3. scripts/ci/reconcile_shards.py (main()) — fully unchanged by this mission, both in shape and in behavior: it gains no retry logic, remains a pure single-pass function over an already-downloaded local directory with no network/API access (FR-005/FR-006). 4. A new Python script/module this mission is expected to add for the artefact-visibility polling step (illustratively, e.g. scripts/ci/wait_for_artifacts.py — the exact name/path is a plan-phase decision, not binding here), invoked from ci-aggregate.yml ahead of its existing actions/download-artifact steps — a net-new entity this mission creates, owning the one retryable network/API call in the ci-aggregate.yml surface (FR-005).
Success Criteria (mandatory)
Measurable Outcomes
- SC-001: Unit tests exist in
tests/ci/covering the surfaces enumerated in NFR-003 —fleet_verdict.py::report(),fleet_main.py::report(), and the newci-aggregate.ymlartefact-visibility polling step/script (mocking racy/changing evidence or artefact-presence sequences, and asserting both the recovery-within-budget path and the retry-budget-exhausted terminal path for each), plus the FR-009 cross-invocation-isolation property.reconcile_shards.py::main()is exercised unchanged by these tests, confirming its existing fail-closed exit and::error::annotation still fire against whatever the (possibly retried) download step actually produced. This is verified pre-merge, by running the tests, not by a green PR. - SC-002: Both safety invariants are verified by the unit tests in SC-001 under simulated instability: (a)
fleet_verdict.py/fleet_main.pynever publish a comment/incident built from a stale (first) snapshot when the two snapshots disagree, even across bounded retries; (b)reconcile_shards.py::main()still exits non-zero with its::error::annotation when a selected/must_be_freshshard remains absent from what the (possibly retried)ci-aggregate.ymlartefact-visibility polling step/download step actually produced, after that polling step's own bounded retry budget is exhausted — the budget belongs to the polling step, not toreconcile_shards.py's own invocation. - SC-003: The existing baseline test suite for these three modules (
.venv/bin/python -m pytest tests/ci/test_fleet_verdict.py tests/ci/test_fleet_main.py tests/ci/test_reconcile_shards.py -q, 98 passed / 0 failed as measured 2026-09-22) continues to pass in full after the fix, with the new tests from SC-001 added alongside — no pre-existing test is broken or deleted to make room for the new behavior. (The unrelated, pre-existing 34 failures intests/ci/test_aggregate_source.py, caused by aModuleNotFoundError: No module named 'coverage'in this checkout's.venv, are an environment gap outside this mission's scope and are explicitly not attributed to this mission if they reappear in a later gate run.) - SC-004 (post-merge observation, not a merge gate): Over a bounded post-merge watch window of N = 20 consecutive
main-head CI cycles, observed over M = 5 calendar days following merge (whichever bound is reached first ends the window), zero same-SHA reconcile flaps (a same-SHA fail-then-pass sequence within the same reconcile step, as documented in #4652/#4675/#4878) are observed across all three job shapes (report-main,CI Aggregatecollect/dedup, per-PRreport (<pr>)). This is the actual confirmation that the race is closed, per Decision 2 — it is explicitly not a condition this mission's own PR must satisfy to be merged, and it is not gated by this mission's PR going green (per C-001/Decision 2, the PR's CI cannot exercise the realworkflow_runpath against its own code at all).