Checklists

requirements.md

Requirements Quality Checklist: Sleep-count assertions survive concurrent sleepers

Purpose: gate spec.md before /spec-kitty.plan. Checks requirement hygiene (IDs, status, measurable thresholds), evidence provenance (no invented numbers), and — the load-bearing one for this programme — that no success criterion is satisfiable by an implementation that fixes nothing. Created: 2026-08-05 · Last run: 2026-08-05 (iteration 3, against spec revision R2) Mission: spec.md · Directive: analysis-report.md

Run log:

CHK021 and CHK026 against a spec whose live evidence was falsified and four of whose criteria were satisfiable by a one-line rewrite. A post-spec adversarial squad found 6 of 13 criteria passes-while-broken and two blockers. The checklist's own miss is recorded below under ## Why iteration 2 passed a spec that was 6-of-13 broken.

because R2's operator rulings changed what the item should assert, and 8 items added (CHK035– CHK042) covering the failure modes iteration 2 had no item for. 3 of the allowed 3 iterations used — no further iteration is available, so the residuals in ## Residual risk are carried to plan rather than fixed here.

  • Iteration 1 — 7 failures (CHK008, CHK013, CHK019, CHK021, CHK024, CHK027, CHK034); fixed.
  • Iteration 2 — all items re-run, clean. This verdict was wrong: it passed CHK012, CHK019,
  • Iteration 3 (this run, against R2) — 8 items re-failed and were fixed, 3 items amended

Structure and identity

  • ✅ CHK001 The three scaffold sections (## User Scenarios & Testing, ## Requirements, ## Success Criteria) appear with unmodified headings, in scaffold order. Verified iteration 3: grep -n '^## ' → 131, 302, 358.
  • ✅ CHK002 Additive sections sit before the first scaffold section (## Problem, line 11) or after the last (## Out of scope 796, ## Corrections 809, ## Unverified 865, ## Open questions 895) — none interleaved.
  • ✅ CHK003 Requirement types are in separate tables with type-unique IDs: FR-001..FR-010, NFR-001..NFR-007, C-001..C-010. No ID reused across types. FR-010 added in R2 for the R-1 alias seam.
  • ✅ CHK004 Every requirement row has a non-empty Status. Measured: 27 rows total (FR 10 + NFR 7 + C 10) — 25 Open, 2 Retired (R-1) (FR-006, FR-007), retained with reasoning and not renumbered, so no successor reads a gap.
  • ✅ CHK005 Every open requirement row has a non-empty Priority. Measured: FR open rows — 6 High / 1 Medium / 1 Low. NFR — 6 High / 1 Medium (NFR-005 Medium; NFR-007 raised from Medium to High in R2, because it is the criterion that makes <census> trustworthy and three criteria depend on <census> alone). C — 8 High / 2 Medium. Amended in iteration 3: the item now scopes to open rows, because a retired requirement carries Priority: — by design. Rewriting a retired row's priority as Low would imply it is still work.
  • ✅ CHK006 Success criteria carry stable IDs SC-001..SC-016 and every delivery criterion maps to at least one FR/NFR/C. SC-001→FR-001/002, SC-002→FR-002, SC-003→FR-004, SC-004→FR-003, SC-005→NFR-001, SC-007→FR-005, SC-008→NFR-004, SC-009→NFR-002/003, SC-010→FR-008, SC-012→NFR-006, SC-013→FR-010, SC-014→NFR-005, SC-015→NFR-007, SC-016→C-002/C-004/C-008/C-010. This also closes the squad's finding that SC-006 was the only criterion with no requirement anchor — it is retired, so no unanchored criterion remains.

Measurability

  • ✅ CHK007 Every NFR states a numeric threshold with a unit: NFR-001 ≥ 100 calls; NFR-002 10/10 runs; NFR-003 6 runs / identical pass sets; NFR-004 0 errors + pin count pinned at 12; NFR-005 ≤ 5.0 s / ≤ 60 s; NFR-006 0 findings, 0 added suppressions, 0 added config ignores; NFR-007 AST-derived + committed control fixture.
  • ✅ CHK008 Every NFR threshold is measurable against a baseline that exists or is required to be captured. Iteration 1: FAIL (NFR-005 had no baseline). Iteration 3: re-failed — NFR-005 still had no criterion at all, and its command was python3 -m pytest, which on this machine is 3.14.4 without project dependencies while CI pins 3.12. Fixed: SC-014 added as NFR-005's criterion; both arms pinned to uv run --python 3.12, with python -V reported per arm and required identical. NFR-007 also had no criterion and <census> is the sole instrument for three criteria — fixed by SC-015.
  • ✅ CHK009 Every delivery criterion is a command plus its expected output, not prose. 14 of them are. Amended in iteration 3: SC-006 (retired) and SC-011 (demoted) are prose entries that explain their own removal from the criterion count; the item now scopes to delivery criteria and CHK038 checks that the exclusions are honest.
  • ✅ CHK010 Expected outputs are exact strings or numeric comparisons, not "should look right". Iteration 3 fix: SC-003's pinned text was [1.35, 3.0, 6.6], which is unsatisfiable — production computes 6.0 1.1 = 6.6000000000000005 (verified this session). An unsatisfiable literal is worse than a vague one: it can never be reported as met, so it gets silently downgraded. Corrected, with the note that the unmutated [0.9, 2.0, 4.4] is exact.*
  • ✅ CHK011 Where a criterion names an artifact this mission will create, the criterion pins the output, not the path. <census>, <guard>, <gate>, <census-control>, <guard-rationale>, <wp-notes> — all placeholders, defined above SC-001. Production symbols named in FR-010 are the R-1 exception, not artifact paths; see CHK026.
  • ✅ CHK042 Every quoted provenance command reproduces its quoted number. NEW in iteration 3, and it fails on R1: spec.md:370-371 quoted grep -c 'patch("specify_cli.tracker.saas_client\.' as the source of the 15-vs-14 pair, and that command returns 68 (verified this session). Fixed: replaced with the command actually run, grep -rc 'patch("specify_cli.tracker.saas_client\.time\.sleep")' <two files> → 14 + 1 = 15. A number with the wrong command attached is indistinguishable from an invented one, which is exactly what CHK012 exists to prevent — so CHK012 was passing on a technicality.

Evidence provenance

  • ✅ CHK012 Every count in the spec traces to a command that was run, or is marked [UNVERIFIED]. Iteration 3: re-failed on three counts — the 15-vs-14 provenance command (see CHK042), NFR-001's "33× above 48" (arithmetically impossible: 100/48 = 2.08, and 33.3 = 100/3 is computed against the expected count, not the observed inflation), and SC-006 Arm A's unattributed 2116 passed, 11 skipped split. All three fixed. Counts re-verified this session by command: 22 files / 14 patch sites (13+1) / 4 nodes / 5 assertions / 53 E-rows / 73 collected / 28-vs-13 / 15-vs-14 / 68 (the wrong command's real answer) / 12 _PinnedLeak( entries (AST) / 6.6000000000000005 / grep -cE over two files printing per-file counts / per-file-ignores present in both config files / upstream/main = 98198e980 / PR #3209 head = 5e98c2bb7 (gh pr view). Six [UNVERIFIED] entries now recorded, up from two.
  • ✅ CHK013 The census probe was controlled against a target whose answer is known independently, and the control is shown. Iteration 1: FAIL. Iteration 3: re-failed for a different reason — the control was described in prose and commanded nowhere, while <census> is the sole instrument for SC-001, SC-002 and SC-013, so a hardcoded output table satisfied all three; and every positive case in it was decorator-shaped, which is why the census missed nine instances. Fixed: SC-015 commits the fixture as a test with ground truth pinned in-test, adds context-manager patch() and side_effect=-kwarg positive cases, and adds a census self-mutation arm.
  • ✅ CHK014 The two code facts the brief asked to verify are verified with the command shown: saas_client.py:19 bare import time; the patch-decorator census.
  • ✅ CHK015 Established findings reused from #3115/#3136 are cited, not re-derived, and marked as such (### Established, reused, not re-derived).
  • ✅ CHK016 Facts the spec declines to trust are named as such. C-007 + FR-008 on the inventory verdict column; six [UNVERIFIED] entries, three of which are squad-sourced CI-log figures this session did not re-fetch (28, 11/18, 39%) with their load-bearingness stated for each.
  • ✅ CHK041 No claim rests on a line number that was not opened. NEW in iteration 3, and the highest-value item in the set — it is the failure mode that produced this whole programme. R1 failed it twice: test_saas_client.py:804 was cited as an assertion when it is a side_effect stimulus (the only assertion is the pytest.raises at :806), and the claim was inherited from a docstring at :55-57. The sibling mission failed it identically at tests/integration/test_coord_loop_workspace.py:611. Iteration 3 opened and read every load-bearing citation: saas_client.py:18-19, :104-106, :439, :478-481, :484, :515-518; test_saas_client.py:24, :55-57, :513, :532, :550, :559, :762, :784, :786, :787, :804-807; test_saas_client_origin.py:22, :229, :261; test_final_sync_diagnostics.py:27, :298-312; test_git_metadata.py:28, :218, :226, :242, :249, :274, :281, :471, :530; sync/batch.py:11; sync/git_metadata.py:13-14; _leak_guard.py:333, :424; conftest.py:461-500; ci-quality.yml:1155-1172; charter.md:504-516.
  • ✅ CHK017 Every file:line citation resolves on 98198e980. Subsumed by CHK041 this iteration, which is strictly stronger — resolving is necessary but not sufficient.

Adversarial soundness — the load-bearing section

  • ✅ CHK018 For every success criterion, an implementation that satisfies it without fixing anything is constructed and recorded. ### Adversarial Analysis of Success Criteria: 40 rows across 16 criteria plus two that apply to all. Rows marked [R2] are the squad's cheats; one row is marked [R2-open] — a cheat acknowledged as not fully closed, with the residual stated rather than hidden.
  • ✅ CHK035 Cardinality is measured, not only required in prose. NEW in iteration 3 — the sharpest hole the squad found. assert 3.0 in [c.args[0] for c in mock_sleep.call_args_list] is honestly non-corruptible, honestly value-checking, satisfies SC-001/SC-002 and both R1 mutation arms, and silently drops cardinality. FR-002 forbade it in prose and measured it nowhere; SC-001 pinned nodes only. Fixed two ways: sleep_assertions: 5 added as an SC-001 denominator, and SC-003 Arm 3 added as a cardinality mutation (duplicate the _sleep(...) call at saas_client.py:439; add a fourth pending response) requiring red on the count with observed-vs-expected counts in the failure text. SC-002 additionally requires n= to be derived from the assertion's cardinality expression, not from the printed list's length.
  • ✅ CHK019 Every criterion whose form is a negative has a positive twin that the instrument demonstrably catches. Iteration 1: FAIL (SC-008, SC-010). Iteration 3: re-failed on five negatives — (a) SC-008's pin grep '^\+._PINNED_LEAKS' is structurally inert: entries are _PinnedLeak(...) calls and the token appears only at _leak_guard.py:333 and :424, so a diff adding a real entry matches nothing (fixed: ^\+\s_PinnedLeak\( plus an AST count pinned at 12); (b) SC-008's twin is serial-only — conftest.py:483-492 prints a different line under xdist and :467-468 documents a real -n 4 run printing inspected 0 test(s) (fixed: -n0 pinned, property stated); (c) SC-012's "0 added # noqa" had a config escape hatch (fixed: config diff reported as text); and two negatives I introduced in this revision and had to twin — (d) SC-007 arm 4's "0 direct calls" (twin: report the 3 alias definitions and 5 rerouted call sites, so a checker that parses nothing fails loudly), (e) SC-016's C-008 "no output" (twin: a sibling git diff --stat on saas_client.py must be non-empty, proving the ref resolves) and SC-013 sub-4's grep → 0 (twin: grep -c 'side_effect stimulus' ≥ 1, so the fix is a correction and not a deletion).
  • ✅ CHK037 Every negative-form pin or diff check is structurally capable of matching what it claims to refuse. NEW in iteration 3, generalising the _PINNED_LEAKS lesson: a check whose pattern cannot match a real violation is worse than absent, because it reports success. Applied to every grep -c … = 0 in the spec: the _PinnedLeak( pattern (fixed), the # noqa diff pattern (matches — ^\+.# noqa does match an added suppression), the E-number patterns (match — verified this session), the is the assertion docstring pattern (matches — returns 1 today, which is the proof it can), and ruff format in WP notes (matches by construction). Each was tested against a case it must refuse, not assumed.*
  • ✅ CHK020 No criterion is satisfiable by a test pass at any breadth. Iteration 2 passed this on the claim that every arm is "an injection or a full-shard composition". The full-shard half is now known to be worthless: pristine main reddens 11 of 18 jobs including at 98198e980, so a clean full shard is the pre-fix outcome 39% of the time. Fixed: every surviving arm is an injection (SC-003/004/005), a static measurement (SC-001/002/007/008/010/012/013/015/016), or a repetition-counted determinism arm (SC-009). The one probabilistic criterion was retired, not weakened.
  • ✅ CHK021 "The victim tests pass" does not appear as a criterion in any form. Iteration 1: FAIL (SC-006 Arm A). Iteration 2 "fixed" it by labelling Arm A a control — which was factually false, since main is not green there. Iteration 3: SC-006 retired in full, both arms, with the reasoning recorded in place and in ## Out of scope. Story 1's acceptance scenario 3 (the shard-composition scenario) likewise withdrawn and replaced with the base-branch red.
  • ✅ CHK022 "No assertion keys on a process-global counter" cannot be satisfied by deleting the assertions. SC-001's four denominators plus sleep_assertions: 5; SC-002's four contracts at the same values with cardinality-derived n=; SC-003's three mutation arms.
  • ✅ CHK023 The red-first proof is a genuine failure of the current assertion shape, not a paraphrase. Iteration 3: re-failed — SC-004 enumerated four pre-fix forms for five census assertions; assert delays == [0.9, 2.0, 4.4] (:786) had no pre-fix red arm and is the member most likely to be cosmetically "hardened" while still reading the unfiltered recorder. Fixed: SC-004 is now five tabulated rows, one per census assertion, each with its literal pre-fix expression, and the guard prints the five row identifiers so a reviewer can count them.
  • ✅ CHK024 The instrument that produces the red is itself proven to be firing. SC-005's mutual dependency with SC-004 arm (b). Both squad lenses attacked this criterion and failed; it is the best-built one in the set and is carried forward substantively unchanged.
  • ✅ CHK025 The gate (FR-005) is non-vacuous. Iteration 3: re-failed on two counts — (a) the gate was keyed on time.sleep, so mock_randbelow.call_count == 1 three lines from a census assertion, and mock_run.call_count == 2, walked past it; (b) its floors were counts (scanned_files >= 22), satisfiable by globbing any 22+ files under tests/ while never opening tests/sync/tracker/. Fixed by R-2: mechanism-keyed predicate, tests/sync/ enforcement, named-file membership sets, the 13 + 1 split, the four node-ids verbatim, three self-mutation arms (decorator / subprocess.run / context-manager-with-side_effect=), and a seam-routing arm.
  • ✅ CHK036 Every floor is a membership set where a count would be gameable. NEW in iteration 3. Applied to SC-007 (names four files, the 13/1 split, four node-ids), SC-004 (five row identifiers), SC-001 (lists the 9 non-tracker instances with dispositions), SC-015 (per-count observed-vs-ground-truth). A floor of the form N >= k is only acceptable where the identity of the counted things is pinned elsewhere.
  • ✅ CHK040 The charter's red-on-base requirement is addressed, or the exception is documented. NEW in iteration 3, and R1 failed it silently: the charter's ATDD-First Discipline (charter.md:504-513) requires the test RED on planning_base_branch and GREEN on the final commit, and R1 mentioned neither (grep -c 'ATDD\|planning_base_branch' spec.md → 0). Injection-red is not base-branch-red — arm (b) is designed to raise and be caught, on any branch. Fixed: ### Charter red-on-base — the exception, named tabulates three arms that are red on 98198e980 for structural reasons, the load-bearing one being that patch("…saas_client._sleep") cannot be set up because the R-1 alias does not exist at that SHA. The AttributeError text is marked [UNVERIFIED] (no test bodies were run); the fact of the red is structural.

Scope discipline

  • [~] CHK026 The spec says WHAT and WHY; no code seam is prescribed. AMENDED — deliberate exception, not a pass. Iteration 2 recorded this as passing and it did. R-1 overrides it: the operator has ruled the product-side module-local alias, so the spec now names _sleep / _monotonic / _randbelow, their five call sites, the ADR, and the _poll_jitter_multiplier resolution (FR-010). Recorded as an operator ruling, bounded to those items, and justified in ## Corrections to the incoming brief #5: the seam's three conditions (ADR, gate arm, dead-seam resolution) are what stop it becoming the next _poll_jitter_multiplier, and none of them is a plan-level choice. The plan retains freedom over <census>, <guard>, <gate> and every artifact path. Marked [~] rather than [x] so the exception stays visible to a reviewer.
  • ✅ CHK027 Operator decisions are treated as settled and not re-opened. Five now: the three from the brief (the lever is the assertion class; the contention reproduction is a non-goal; the inventory verdict column is unverified) plus R-1 and R-2 from the post-spec squad. All five named as settled where they appear.
  • ✅ CHK028 At most 3 clarification markers. Now 0 — all three R1 markers settled by the rulings and recorded struck with resolutions. Not asserted as a grep: any command quoting the marker string matches its own quotation.
  • ✅ CHK029 Each surviving clarification marker is a decision the operator must make, not a gap the plan can close. Vacuously satisfied (none survive), and each retirement records why it was an operator decision and how it was decided — including that marker #2 carried a false blocker (301.0 "IS its assertion"), and that deferring on a non-existent blocker would not have been a legitimate decision.
  • ✅ CHK030 Non-goals are recorded with reasoning, not just listed. ## Out of scope — explicitly, 8 entries (up from 6). The contention reproduction carries all three reasons and is duplicated into C-005 so it survives a partial read; SC-006's retirement carries its full derivation; the acknowledged residual (seam check not widened beyond saas_client.py) is listed rather than hidden.
  • ✅ CHK031 Constraints the brief specified are all present: tests/sync/tests/cli never concurrent and window-holder check (C-001); ruff check only (C-002); #3130 not fixed here (C-003).
  • ✅ CHK039 Every constraint has an enforcement check, and every command is runnable in the target environment. NEW in iteration 3, failing on both halves. (a) C-002, C-004, C-008, C-010 had no enforcement anywhere, two of them High — fixed by SC-016, one command each, with C-004 restated so it can actually fail (R1's form rested on a git checkout -- revert, which a clean revert satisfies even if a different line shipped changed; and under R-1 its implied git diff … empty check is false by construction). (b) every SC command was python3 -m pytest, unrunnable here: the mission venv is Python 3.11.15 with no pytest and no ruff, the ambient interpreter is 3.14.4, and CI pins 3.12 — three interpreters, none of them the right one. Fixed: all commands are uv run --python 3.12 python … (uv verified present), and the first WP provisions the environment and records the resolved versions before any acceptance arm runs.
  • ✅ CHK032 Findings that exceed the brief are folded in rather than silently dropped. Six entries in ## Corrections to the incoming brief, two of which correct this spec's own R1 — the withdrawn composition claim and the un-prescribed-seam exception — plus the docstring-as-constraint failure mode recorded as a programme-level rule.
  • ✅ CHK038 The criterion count is honest: retired and demoted criteria are excluded from it and say why. NEW in iteration 3. SC-011 was vacuous — measured 3 / 3 / 5 against the spec as committed, before any work, so it graded the spec's prose and no implementation could fail it. SC-006 discriminates nothing. Both are retained as IDs with in-place explanations and excluded from the stated count, which is now "14 delivery criteria: SC-001–005, SC-007–010, SC-012–016". Renumbering was rejected: a successor reading SC-006 in the plan or a WP must find it, not find a gap.
  • ✅ CHK033 The spec keys on the class, and where it lists node-ids it says why the list is safe to state. Strengthened in R2: the class is now defined by the mechanism predicate (penultimate segment resolves to a ModuleType) as a Key Entity, with the census demoted to a derived instance list. R1's claim that the class is "confined to tests/sync/tracker/" is struck — it was a statement about R1's census, which inspected decorators only and could not match a side_effect= kwarg, and nine further instances under tests/sync/ were each opened and read.
  • ✅ CHK034 No two requirements read as contradicting each other. Iteration 1: FAIL (FR-001 vs FR-002). Iteration 3: re-failed — C-004 ("no change to saas_client.py") directly contradicts FR-010 (change saas_client.py). Fixed: C-004 restated as "changed only by the declared alias seam and the _poll_jitter_multiplier resolution, with the retry behaviour unchanged", with SC-016 enumerating the permitted hunks. One numeric disagreement is left standing deliberately and with an owner: C-003 says "11 confirmed leaks" and the registry holds 12 (_leak_guard.py:333, AST-measured). SC-008 pins 12 and requires the WP to state which reading of 11 was wrong. Two numbers coexisting with a named reconciliation owner is honest; two numbers coexisting silently is what CHK034 exists to catch.

Why iteration 2 passed a spec that was 6-of-13 broken

Recorded because a checklist that cannot explain its own miss will repeat it.

1. *It checked that a cheat was constructed, not that the closure held. CHK018 counted 22 cheats and passed. It had no item asking whether the recorded closure actually closes — and four of them did not, all against the same one-line in-form rewrite. CHK035 now targets that specific class; CHK037 generalises it. 2. It trusted the spec's own live evidence. CHK012 checks that counts trace to a command. R1's 2116 passed, EXIT=0 did trace to a command — it just was not representative, and CI's own logs said the opposite at the same SHA. No item asked whether a measurement's variance had been characterised before it was used as a control. That is why CHK020 now refuses a pass at any breadth rather than trusting the "full-shard composition" escape. 3. It treated a passing grep as a satisfied check. CHK019's iteration-2 fix added two positive twins, and one of them (_PINNED_LEAKS) was a pattern that cannot match a real violation. CHK037 now tests every negative pattern against a case it must refuse. 4. It read line numbers instead of lines. CHK017 asked whether citations resolve*. Every one of R1's did. test_saas_client.py:804 resolves and is not an assertion. CHK041 replaces resolution with reading, and it is the item most worth a reviewer's own hand-check. 5. It had no environment item at all. Every one of the 13 criteria was unrunnable on this machine and nothing noticed, because "is this command runnable here" was not a question the checklist asked. CHK039.

Residual risk — carried to plan, not fixed here

The iteration budget is exhausted (3 of 3). These are known and deliberately not closed:

the tracker cone is not covered by SC-007 arm 4, whose AST check is scoped to saas_client.py. Widening the seam check to src/specify_cli/ is left to a successor and is recorded in ## Out of scope.

they support a criterion retirement, which fails safe — but NFR-001's 3.57× ratio depends on 28, and a reviewer who wants that number should re-fetch the logs.

were each opened and read, so they are verified as instances; no command has produced a census total over the widened scope, because <census> does not exist yet. SC-001's non-tracker listing is therefore a requirement on the artifact, not a reproduction of a measured figure.

WP (SC-008) rather than resolved here, because resolving it requires deciding which of the 12 pins is not a #3130 leak — a judgement about #3130's scope, which C-003 puts outside this mission.

  • The [R2-open] row in the adversarial table. A production call added to a different module in
  • Three squad-sourced CI figures are [UNVERIFIED] (28, 11/18, 39%). None gates the mission —
  • The committed census's own total over tests/sync/ is unverified. The nine non-tracker instances
  • C-003's "11" versus the registry's "12" is left as a named reconciliation for the implementing