<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xml:base="https://docs.spec-kitty.ai/adr/4.x/2026-10-15-1-qa-and-core-share-the-acceptance-scenarios-and-absorb-engineering-tests-through-a-ledger/">
  <id>https://docs.spec-kitty.ai/adr/4.x/2026-10-15-1-qa-and-core-share-the-acceptance-scenarios-and-absorb-engineering-tests-through-a-ledger/</id>
  <title>ADR: QA and core share ownership of the acceptance scenarios, and engineering tests move into them only through a checked ledger</title>
  <updated>2026-10-15T00:00:00.000Z</updated>
  <author><name>Spec Kitty</name></author>
  <link rel="alternate" type="text/html" href="https://docs.spec-kitty.ai/adr/4.x/2026-10-15-1-qa-and-core-share-the-acceptance-scenarios-and-absorb-engineering-tests-through-a-ledger/"/>
  <summary>QA and core co-own tests/scenarios and core owns the runner; integration and e2e tests retire into gating scenarios via a checked ledger, and red nights escalate.</summary>
  <content type="html">&lt;p&gt;&lt;strong&gt;Status:&lt;/strong&gt; Proposed, 2026-10-15. Needs the owner’s acceptance. Decision 5 also needs the agreement of the spec-kitty-qa lead, because it settles open item 3 of spec-kitty-qa ADR-0001.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Date:&lt;/strong&gt; 2026-10-15&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Deciders:&lt;/strong&gt; Stijn Dejongh (owner), with the spec-kitty-qa maintainers.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Technical Story:&lt;/strong&gt; epic &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/6101&quot;&gt;#6101&lt;/a&gt;. Follows &lt;a href=&quot;/adr/4.x/2026-10-11-3-acceptance-testing-and-bdd-support/&quot;&gt;ADR 2026-10-11-3&lt;/a&gt; and &lt;a href=&quot;/adr/4.x/2026-10-13-1-acceptance-scenarios-link-to-promises-by-identifier-only/&quot;&gt;ADR 2026-10-13-1&lt;/a&gt;. The plan it governs is the &lt;a href=&quot;/development/testing/test-pyramid-rebalance/&quot;&gt;test-pyramid rebalance plan&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Reader:&lt;/strong&gt; a maintainer or QA engineer who needs to know who owns an acceptance scenario, how a scenario reaches &lt;code dir=&quot;auto&quot;&gt;main&lt;/code&gt;, and when an engineering test may be deleted because a scenario now checks the same thing.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id=&quot;context-and-problem-statement&quot;&gt;Context and Problem Statement&lt;/h2&gt;
&lt;p&gt;The acceptance system exists: Gherkin scenarios in &lt;code dir=&quot;auto&quot;&gt;tests/scenarios/&lt;/code&gt;, the Cucumber-JVM runner in &lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/&lt;/code&gt;, a per-pull-request contract check and the nightly &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd&lt;/code&gt; job on Linux, macOS and Windows. Next to it, &lt;code dir=&quot;auto&quot;&gt;tests/integration&lt;/code&gt; (177 files, 64,552 lines) and &lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt; (7 files, 3,264 lines) carry most of the end-to-end coverage, run only in the nightly (&lt;code dir=&quot;auto&quot;&gt;integration-next&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;e2e&lt;/code&gt;) and name no promise.&lt;/p&gt;
&lt;p&gt;Four questions were unanswered:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Ownership.&lt;/strong&gt; spec-kitty-qa ADR-0001 says QA owns scenarios and engineers own step code, but nothing in this repository says so. &lt;code dir=&quot;auto&quot;&gt;.github/CODEOWNERS&lt;/code&gt; gives every path to one maintainer.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Absorption.&lt;/strong&gt; Nothing defines when an integration or e2e test may be deleted because a scenario covers it. Without a rule, a deleted test is coverage lost without trace, and a duplicated one is cost paid twice.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Signal.&lt;/strong&gt; A red &lt;code dir=&quot;auto&quot;&gt;integration-next&lt;/code&gt; or &lt;code dir=&quot;auto&quot;&gt;e2e&lt;/code&gt; night opens a deduplicated P0 issue (&lt;code dir=&quot;auto&quot;&gt;scripts/ci/nightly_escalation.py&lt;/code&gt;). A red &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd&lt;/code&gt; night fails the run, but files nothing, so coverage moved into a scenario would lose its escalation.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pyramid.&lt;/strong&gt; No document says which layer a test belongs in.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The audit behind this ADR classified all 184 files (the &lt;a href=&quot;/development/testing/integration-e2e-inventory/&quot;&gt;inventory&lt;/a&gt;). Most need internals (monkeypatching, fault injection, hand-made internal state) and stay pytest. A minority can be checked black-box today. Much of the rest needs a Mission with finalized work packages, which needs authored content (spec, tasks and work-package files), which ADR 2026-10-13-1 decision 5 does not allow in a builder.&lt;/p&gt;
&lt;h2 id=&quot;decision-drivers&quot;&gt;Decision Drivers&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;No net loss of coverage: every retired test is accounted for, and the account is checked by a machine on every pull request.&lt;/li&gt;
&lt;li&gt;No feedback moves later: only tests whose signal already arrives nightly may move into the nightly scenarios.&lt;/li&gt;
&lt;li&gt;One owner per kind of artefact, and a review flow that asks the right people.&lt;/li&gt;
&lt;li&gt;Scenarios stay black-box (ADR 2026-10-13-1): steps drive the built CLI as a process.&lt;/li&gt;
&lt;li&gt;Gates that can fail: every new rule has a planted case that must fail it (charter Standing Order 5).&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;considered-options&quot;&gt;Considered Options&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Shared ownership, a checked absorption ledger, nightly-only eligibility, one escalation&lt;/strong&gt; (chosen).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Move all end-to-end coverage to spec-kitty-qa.&lt;/strong&gt; Rejected: a core change could no longer fix its own failing test in the same pull request (spec-kitty-qa ADR-0001 rejected the same move for the nightly).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Delete integration tests on review judgement alone, without a ledger.&lt;/strong&gt; Rejected: nothing would show that a deleted file was absorbed rather than dropped.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tag Python tests with the scenario that absorbs them, and keep both.&lt;/strong&gt; Rejected: ADR 2026-10-11-3 says a test is either a scenario or an engineering test, never both, and keeping both pays for the coverage twice.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Escalate once per operating system.&lt;/strong&gt; Rejected: a product break usually turns every leg red and would file three P0 issues; the escalation script also needs a bare &lt;code dir=&quot;auto&quot;&gt;python3&lt;/code&gt;, which the Windows leg does not have.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;decision-outcome&quot;&gt;Decision Outcome&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Chosen option:&lt;/strong&gt; option 1.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Ownership boundary.&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Shared by QA and core:&lt;/strong&gt; the scenarios (&lt;code dir=&quot;auto&quot;&gt;tests/scenarios/**&lt;/code&gt;, including the absorption ledger), the &lt;a href=&quot;/development/testing/acceptance-scenarios/&quot;&gt;scenario-writing guide&lt;/a&gt; and the &lt;a href=&quot;/development/testing/test-pyramid-rebalance/&quot;&gt;rebalance plan&lt;/a&gt;. &lt;code dir=&quot;auto&quot;&gt;.github/CODEOWNERS&lt;/code&gt; names &lt;code dir=&quot;auto&quot;&gt;@stijn-dejongh&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;@kentonium3&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;@Ivan-Melck&lt;/code&gt; for them.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Core engineers:&lt;/strong&gt; the runner, step definitions, driver and fixture builders (&lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/**&lt;/code&gt;), named &lt;code dir=&quot;auto&quot;&gt;@stijn-dejongh&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;spec-kitty-qa:&lt;/strong&gt; promise pages, the promise index, the trace lint and the Allure report (ADR 2026-10-13-1).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_scenario_contract.py&lt;/code&gt; checks these CODEOWNERS rules with the last-matching-rule semantics of &lt;code dir=&quot;auto&quot;&gt;contracts/tools/codeowners_check.py&lt;/code&gt;, so a later rule cannot silently take a path away. GitHub review through CODEOWNERS stays advisory; making code-owner review required is a repository setting the operator decides.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Authoring and review flow.&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;QA drafts a scenario from a promise, by hand or with AI help, and opens a pull request here. A drafted scenario lands as &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; with &lt;code dir=&quot;auto&quot;&gt;@wip-owner&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;@wip-since&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;An engineer binds its steps and adds a positive control.&lt;/li&gt;
&lt;li&gt;The &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; tag is removed in a pull request that both a QA owner and a core owner reviewed.&lt;/li&gt;
&lt;li&gt;An engineer who absorbs an engineering test writes the scenario untagged, and asks QA to review its wording.&lt;/li&gt;
&lt;li&gt;The &lt;code dir=&quot;auto&quot;&gt;When&lt;/code&gt; wording question (#6098) is not decided here; scenarios keep the generic &lt;code dir=&quot;auto&quot;&gt;When I run &quot;&amp;#x3C;command line&gt;&quot;&lt;/code&gt; step until it is.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Absorption rule.&lt;/strong&gt; An engineering test from &lt;code dir=&quot;auto&quot;&gt;tests/integration&lt;/code&gt; or &lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt; is retired only in the pull request that adds its replacement, and only with a row in &lt;code dir=&quot;auto&quot;&gt;tests/scenarios/absorption-ledger.json&lt;/code&gt;. The replacement is either a gating scenario that asserts at least what the test asserted, on an equal or stronger arrange, or, for a duplicate, the surviving pytest tests that already do.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The row’s &lt;code dir=&quot;auto&quot;&gt;note&lt;/code&gt; says what changed in the arrange and what the replacement does not cover.&lt;/li&gt;
&lt;li&gt;The ledger freezes a baseline: the 184 test files of the two trees on 2026-10-15.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;scripts/ci/absorption_ledger.py&lt;/code&gt; runs on every pull request through &lt;code dir=&quot;auto&quot;&gt;test_scenario_contract.py&lt;/code&gt;. It fails when:
&lt;ul&gt;
&lt;li&gt;a retired test still exists;&lt;/li&gt;
&lt;li&gt;a replacement scenario is missing, &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; or &lt;code dir=&quot;auto&quot;&gt;@manual&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;a replacement pytest test is missing;&lt;/li&gt;
&lt;li&gt;a row is duplicated, or names a file outside the baseline;&lt;/li&gt;
&lt;li&gt;a baseline file disappears without a whole-file row.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Reviewers check that the assertions carried over; the machine cannot.&lt;/li&gt;
&lt;li&gt;Removing one test function from a kept file without a row is not detected. That gap is accepted, and reviewers check it.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Eligibility.&lt;/strong&gt; Only tests whose feedback already arrives nightly may be absorbed:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;tests/integration&lt;/code&gt;, which runs only in &lt;code dir=&quot;auto&quot;&gt;integration-next&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt;, which runs on a pull request only when its own files change.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A test that runs on every pull request is not absorbed: moving it would delay its signal to the next morning. It may still gain a scenario next to it when the scenario checks a promise.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Signal parity.&lt;/strong&gt; The new &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd-escalation&lt;/code&gt; job in &lt;code dir=&quot;auto&quot;&gt;ci-nightly.yml&lt;/code&gt; turns a red &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd&lt;/code&gt; run on &lt;code dir=&quot;auto&quot;&gt;main&lt;/code&gt; into one deduplicated P0 issue under the suite key &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd&lt;/code&gt;. It closes the issue on the next green run. Both jobs are in &lt;code dir=&quot;auto&quot;&gt;nightly-summary.needs&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This &lt;strong&gt;amends ADR 2026-10-11-3&lt;/strong&gt;, which treated the core job as an artefact producer only. It also &lt;strong&gt;proposes an answer to open item 3 of spec-kitty-qa ADR-0001&lt;/strong&gt; (issue routing): a red acceptance run files into this repository under that one key, and the spec-kitty-qa acceptance nightly reuses the key rather than opening a second issue for the same red. If spec-kitty-qa disagrees, only the key and the owner of the filing change; the escalation stays.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Authored content in Arrange (decided, operator ruling 2026-10-15).&lt;/strong&gt; A scenario builder may write
the files an agent authors during &lt;code dir=&quot;auto&quot;&gt;specify&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;plan&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;tasks&lt;/code&gt; (&lt;code dir=&quot;auto&quot;&gt;spec.md&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;plan.md&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tasks.md&lt;/code&gt;,
&lt;code dir=&quot;auto&quot;&gt;tasks/WP*.md&lt;/code&gt;), the code a work package’s implementer writes in its checkout, a reviewer’s
feedback file outside the repository, and plain files a user edits in git (a tracked file, an
untracked stray file). It still never writes CLI-managed state: &lt;code dir=&quot;auto&quot;&gt;meta.json&lt;/code&gt;, the status log,
&lt;code dir=&quot;auto&quot;&gt;lanes.json&lt;/code&gt;, review-cycle records, or anything under &lt;code dir=&quot;auto&quot;&gt;.kittify/&lt;/code&gt;. This &lt;strong&gt;amends decision 5 of
ADR 2026-10-13-1&lt;/strong&gt;, which allowed no file writes at all; its purpose (a builder breaks when the
CLI’s own path breaks) holds, because every state change still goes through the CLI. Planning
happens on a topic branch, because Spec Kitty refuses planning commits on the protected primary
branch, which is also what a user does.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;consequences&quot;&gt;Consequences&lt;/h3&gt;
&lt;h4 id=&quot;positive&quot;&gt;Positive&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;Every retired test has a checked row; a deletion without one fails the pull request.&lt;/li&gt;
&lt;li&gt;Absorbed coverage runs against the built wheel on three operating systems instead of one, and escalates like the suite it left.&lt;/li&gt;
&lt;li&gt;A scenario change asks QA and core for review, and the ownership rule cannot drift silently.&lt;/li&gt;
&lt;/ul&gt;
&lt;h4 id=&quot;negative&quot;&gt;Negative&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;Contributors deleting or renaming a baseline file must add a ledger row.&lt;/li&gt;
&lt;li&gt;The ledger only grows; it is the history of the move.&lt;/li&gt;
&lt;li&gt;Many integration tests rightly stay pytest: every baseline file now has a recorded decision, and the move took &lt;code dir=&quot;auto&quot;&gt;tests/integration&lt;/code&gt; from 177 to 151 baseline files (64,552 to 58,396 lines) and &lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt; from 7 to 4, with 56 gating scenarios. Three org-pack files wait on &lt;a href=&quot;https://github.com/spec-kitty/spec-kitty/issues/6267&quot;&gt;#6267&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h4 id=&quot;neutral&quot;&gt;Neutral&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;New tests added to &lt;code dir=&quot;auto&quot;&gt;tests/integration&lt;/code&gt; or &lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt; after the baseline are not tracked by it.&lt;/li&gt;
&lt;li&gt;The release gate already read the whole nightly run, so a red acceptance run already blocked a release; only the escalation is new.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;confirmation&quot;&gt;Confirmation&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_scenario_contract.py&lt;/code&gt; passes on the real tree, and each planted ledger fails its own rule.&lt;/li&gt;
&lt;li&gt;The same file fails when a planted CODEOWNERS line takes &lt;code dir=&quot;auto&quot;&gt;tests/scenarios/&lt;/code&gt; away from the shared owners.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;tests/ci/test_acceptance_bdd_job.py&lt;/code&gt; pins one escalation job with one key, fail-closed, mainline-only, aggregated by &lt;code dir=&quot;auto&quot;&gt;nightly-summary&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;For every absorbing scenario, putting the guarded defect back turned it red and left the others green: the #4754 abort, sparse-checkout detection, the sparse-checkout preflight of &lt;code dir=&quot;auto&quot;&gt;implement&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;consolidate&lt;/code&gt;, the #4863 no-state abort, the #3716 discard provenance, the #840 schema stamp, the not-initialised gate, the single-branch review commit gate, the empty-feedback refusal (both layers), the declared reviewer, the occupied and dirty write checkout, the kept ref of a discarded Mission branch, the setup-plan spec gate, the frozen lane membership, the synthesize dry run, the missing-bundle remedy, the synthesize remediation and the no-charter detail.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/build.gradle&lt;/code&gt; declares the test task never up to date, because the wheel under test is built at run time from a checkout Gradle does not track; without that, a re-run after a source change reported the previous results.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;more-information&quot;&gt;More Information&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;/development/testing/test-pyramid-rebalance/&quot;&gt;Test-pyramid rebalance plan&lt;/a&gt;: layers, decision rules, waves and metrics.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;/development/testing/integration-e2e-inventory/&quot;&gt;Integration and e2e inventory&lt;/a&gt;: the per-file audit of 2026-10-15.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;/development/testing/acceptance-scenarios/&quot;&gt;CLI acceptance scenarios&lt;/a&gt;: how to write, run and absorb.&lt;/li&gt;
&lt;/ul&gt;</content>
</entry>
