<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xml:base="https://docs.spec-kitty.ai/adr/4.x/2026-10-11-3-acceptance-testing-and-bdd-support/">
  <id>https://docs.spec-kitty.ai/adr/4.x/2026-10-11-3-acceptance-testing-and-bdd-support/</id>
  <title>ADR: acceptance testing and BDD support with Gherkin scenarios and a black-box Cucumber-JVM runner</title>
  <updated>2026-10-11T00:00:00.000Z</updated>
  <author><name>Spec Kitty</name></author>
  <link rel="alternate" type="text/html" href="https://docs.spec-kitty.ai/adr/4.x/2026-10-11-3-acceptance-testing-and-bdd-support/"/>
  <summary>Promise-tagged Gherkin scenarios live in tests/scenarios, a per-PR Python check guards their tags, and a nightly Cucumber-JVM job drives the built CLI as a black box.</summary>
  <content type="html">&lt;p&gt;&lt;strong&gt;Status:&lt;/strong&gt; Accepted — Amended 2026-10-11 — see &lt;a href=&quot;#amendment-2026-10-11-owner-ruling&quot;&gt;Amendment&lt;/a&gt;; the promise-machinery paragraphs below are narrowed by it (never built in core).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Date:&lt;/strong&gt; 2026-10-11&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Deciders:&lt;/strong&gt; Stijn Dejongh (owner), accepted 2026-10-11.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Technical Story:&lt;/strong&gt; the QA promise train (spec-kitty-planning &lt;code dir=&quot;auto&quot;&gt;qa-promise-train/&lt;/code&gt;). Its QA half is spec-kitty-qa ADR-0001 (QA owns the acceptance nightly and the Allure report, amended 2026-10-11) and ADR-0002 (a promise index joins the promise pages to core’s scenarios). This ADR is the core half.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Reader:&lt;/strong&gt; a maintainer who needs to know where acceptance scenarios live, what checks them on a pull request, how the nightly runs them, and why the runner is Java while the CLI stays Python.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id=&quot;context-and-problem-statement&quot;&gt;Context and Problem Statement&lt;/h2&gt;
&lt;p&gt;Spec Kitty promises behaviour to its users (for example “re-running &lt;code dir=&quot;auto&quot;&gt;init&lt;/code&gt; keeps project state”, from &lt;code dir=&quot;auto&quot;&gt;init --help&lt;/code&gt;), but nothing ties a promise to a test that checks it. Core has about 230 Python acceptance and e2e tests (&lt;code dir=&quot;auto&quot;&gt;tests/acceptance&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt;), and the nightly runs them, but none of them names the promise it protects, so a release cannot say which promises it verified. The CLI’s own spec template already asks for Given/When/Then, yet core has no &lt;code dir=&quot;auto&quot;&gt;.feature&lt;/code&gt; files and no BDD runner.&lt;/p&gt;
&lt;p&gt;The product owner decided on 2026-10-11, after a four-lens design review:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Promise pages stay private in spec-kitty-qa; Gherkin scenarios live in this repository; a synced promise index joins them.&lt;/li&gt;
&lt;li&gt;The runner is Cucumber-JVM, chosen for cross-OS stability and because parts of core (the Mission API, the charter offering and activation) are planned to move to Java modules.&lt;/li&gt;
&lt;li&gt;A scenario whose steps are not yet bound lands as &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; with an owner and an age limit, and does not gate a release.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;considered-options&quot;&gt;Considered Options&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;pytest-bdd inside the existing Python suite.&lt;/strong&gt; Rejected by the product owner in favour of a JVM runner; recorded as rejected in spec-kitty-qa ADR-0001.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Scenarios in spec-kitty-qa.&lt;/strong&gt; Rejected: an operator could not open a scenario pull request against the code it describes, and a bug reproduction would not share history with its fix.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Promise pages in this repository too.&lt;/strong&gt; Rejected: candidate promise wording would be public.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Java steps that call core’s Python or future Java classes directly.&lt;/strong&gt; Rejected: ADR 2026-10-01-2 keeps the CLI native and ADR 2026-10-03-1 keeps writes in Python during the strangler, so steps drive the CLI as a process and stay valid on either side.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;decision&quot;&gt;Decision&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Scenarios.&lt;/strong&gt; Gherkin files live under &lt;code dir=&quot;auto&quot;&gt;tests/scenarios/&amp;#x3C;area&gt;/&lt;/code&gt;. A scenario cites the promises it checks with &lt;code dir=&quot;auto&quot;&gt;@promise:&amp;#x3C;id&gt;&lt;/code&gt;. Other tags: &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; with &lt;code dir=&quot;auto&quot;&gt;@wip-owner:&amp;#x3C;handle&gt;&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;@wip-since:&amp;#x3C;YYYY-MM-DD&gt;&lt;/code&gt; for unbound steps, &lt;code dir=&quot;auto&quot;&gt;@defect:&amp;#x3C;issue&gt;&lt;/code&gt; for a bug reproduction, &lt;code dir=&quot;auto&quot;&gt;@qa:SC-A-NNN&lt;/code&gt; for a scenario converted from the QA corpus. &lt;code dir=&quot;auto&quot;&gt;tests/scenarios/promise-index.json&lt;/code&gt; is the public slice of spec-kitty-qa’s promise index (ratified and retired ids with titles, &lt;code dir=&quot;auto&quot;&gt;schema_version: 1&lt;/code&gt;). It arrives only through a pull request opened by an org-owned token and is never edited by hand.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Per pull request.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_scenario_promise_tags.py&lt;/code&gt;, stdlib only, asserts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;a scenario without &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; cites at least one &lt;code dir=&quot;auto&quot;&gt;@promise:&amp;#x3C;id&gt;&lt;/code&gt;, and each id is &lt;code dir=&quot;auto&quot;&gt;ratified&lt;/code&gt; in the index;&lt;/li&gt;
&lt;li&gt;a &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; scenario may cite an id the index does not have yet (a candidate);&lt;/li&gt;
&lt;li&gt;no scenario cites a &lt;code dir=&quot;auto&quot;&gt;retired&lt;/code&gt; id;&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; carries its owner and since tags, and &lt;code dir=&quot;auto&quot;&gt;@defect&lt;/code&gt; carries an issue number.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The test carries one broken fixture per rule. &lt;code dir=&quot;auto&quot;&gt;.github/workflows/ci-router.yml&lt;/code&gt; gains a &lt;code dir=&quot;auto&quot;&gt;scenarios&lt;/code&gt; group (&lt;code dir=&quot;auto&quot;&gt;tests/scenarios/**&lt;/code&gt;) that gates the architectural job, and an &lt;code dir=&quot;auto&quot;&gt;acceptance_jvm&lt;/code&gt; group (&lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/**&lt;/code&gt;) that gates no per-PR job, like the &lt;code dir=&quot;auto&quot;&gt;ci&lt;/code&gt; group. Contributors need no JDK.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Nightly runner.&lt;/strong&gt; A Gradle subproject at &lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/&lt;/code&gt; (the JDK core’s Java modules use, Java 25 per the 2026-10-03 amendment of ADR 2026-10-01-2; pinned Gradle distribution, no committed wrapper jar, dependency verification metadata, following &lt;code dir=&quot;auto&quot;&gt;contracts/&lt;/code&gt;) runs the scenarios in place from &lt;code dir=&quot;auto&quot;&gt;../tests/scenarios&lt;/code&gt; with Cucumber-JVM 7 on the JUnit Platform and &lt;code dir=&quot;auto&quot;&gt;allure-cucumber7-jvm&lt;/code&gt;.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;A &lt;code dir=&quot;auto&quot;&gt;SpecKittyDriver&lt;/code&gt; port runs a command and returns exit code, output, parsed &lt;code dir=&quot;auto&quot;&gt;--json&lt;/code&gt; and the working tree. &lt;code dir=&quot;auto&quot;&gt;CliProcessDriver&lt;/code&gt; implements it with &lt;code dir=&quot;auto&quot;&gt;ProcessBuilder&lt;/code&gt;: drained streams, a timeout, descendant processes killed, line endings normalised, real paths compared.&lt;/li&gt;
&lt;li&gt;Each scenario gets a temporary git repository and an isolated environment: &lt;code dir=&quot;auto&quot;&gt;SPEC_KITTY_HOME&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;HOME&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;USERPROFILE&lt;/code&gt;, the XDG directories and &lt;code dir=&quot;auto&quot;&gt;LOCALAPPDATA&lt;/code&gt; inside it, UTF-8, &lt;code dir=&quot;auto&quot;&gt;NO_COLOR&lt;/code&gt; and a fixed &lt;code dir=&quot;auto&quot;&gt;COLUMNS&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The CLI under test is a wheel built from the commit (&lt;code dir=&quot;auto&quot;&gt;uv build&lt;/code&gt;) and installed into a temporary venv; never PyPI, never an editable install.&lt;/li&gt;
&lt;li&gt;Steps call only the driver. They never import Python and never use internal Java classes; published contracts (for example &lt;code dir=&quot;auto&quot;&gt;contracts/mission-status/openapi.yaml&lt;/code&gt;) are allowed.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A new &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd&lt;/code&gt; job in &lt;code dir=&quot;auto&quot;&gt;ci-nightly.yml&lt;/code&gt; runs on Linux, macOS and Windows. The gating run filters &lt;code dir=&quot;auto&quot;&gt;not @wip and not @manual&lt;/code&gt;; a separate non-gating run executes bound &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; scenarios. No retries. Results (Allure, Cucumber JSON, JUnit XML) are uploaded for the spec-kitty-qa acceptance nightly, which runs the promise trace lint and builds the Allure 3 report.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Packaging.&lt;/strong&gt; The wheel ships only the &lt;code dir=&quot;auto&quot;&gt;src/&lt;/code&gt; packages and is unaffected. The sdist &lt;code dir=&quot;auto&quot;&gt;exclude&lt;/code&gt; gains &lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/**&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;tests/scenarios/**&lt;/code&gt;, asserted in &lt;code dir=&quot;auto&quot;&gt;tests/cross_cutting/packaging/test_packaging_safety.py&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;One source of promise coverage.&lt;/strong&gt; Only &lt;code dir=&quot;auto&quot;&gt;@promise&lt;/code&gt;-tagged scenarios count as promise coverage. &lt;code dir=&quot;auto&quot;&gt;tests/acceptance&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;tests/e2e&lt;/code&gt;, performance, stress and &lt;code dir=&quot;auto&quot;&gt;p0_repro&lt;/code&gt; stay engineering tests. A Python test that already checks a promise either becomes a scenario or stays untagged, never both.&lt;/p&gt;
&lt;h2 id=&quot;consequences&quot;&gt;Consequences&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;The release robot needs the core nightly and the spec-kitty-qa acceptance nightly green on the same commit (spec-kitty-qa ADR-0001).&lt;/li&gt;
&lt;li&gt;A breaking CLI change to a promised behaviour turns the next nightly red on three operating systems, not only Linux.&lt;/li&gt;
&lt;li&gt;Until spec-kitty-qa decides who ratifies a promise (QAD-0009), the index has no rows and only &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; scenarios can land.&lt;/li&gt;
&lt;li&gt;The same scenarios can later run against a Java implementation by pointing the driver port at it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;open-questions&quot;&gt;Open Questions&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;The org-owned sync secret (&lt;code dir=&quot;auto&quot;&gt;PROMISE_INDEX_SYNC_TOKEN&lt;/code&gt;) still has to be created, scoped like &lt;code dir=&quot;auto&quot;&gt;CHARTER_SHARD_RECAPTURE_TOKEN&lt;/code&gt; to pull requests on this repository. &lt;em&gt;(Narrowed out of core — see the Amendment below; this open question is now closed for core and owned by spec-kitty-qa.)&lt;/em&gt;&lt;/li&gt;
&lt;li&gt;The &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; age limit (30 days is a placeholder) is set in the trace lint, not here.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;amendment-2026-10-11-owner-ruling&quot;&gt;Amendment (2026-10-11 owner ruling)&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Author:&lt;/strong&gt; Stijn Dejongh (owner). &lt;strong&gt;Date:&lt;/strong&gt; 2026-10-11.&lt;/p&gt;
&lt;p&gt;After acceptance, the owner ruled that the promise-tracking machinery this ADR
specified belongs to &lt;strong&gt;spec-kitty-qa&lt;/strong&gt;, not to core. The ruling does not rewrite the
decision above — the Context, Considered Options, Decision and Consequences stay as the
record of what was accepted — it &lt;strong&gt;narrows&lt;/strong&gt; the parts that were specified for core but
&lt;strong&gt;never built here&lt;/strong&gt;. The following four items are owned by spec-kitty-qa (its ADR-0001
and ADR-0002 are the owning home) and were never added to this repository:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;&lt;code dir=&quot;auto&quot;&gt;tests/scenarios/promise-index.json&lt;/code&gt;&lt;/strong&gt; — the public promise-index slice (ratified and
retired ids with titles, &lt;code dir=&quot;auto&quot;&gt;schema_version: 1&lt;/code&gt;). It does not exist in this tree and will not
be added here.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The per-PR check &lt;code dir=&quot;auto&quot;&gt;tests/architectural/test_scenario_promise_tags.py&lt;/code&gt;&lt;/strong&gt; — the stdlib
check that asserted each &lt;code dir=&quot;auto&quot;&gt;@promise:&amp;#x3C;id&gt;&lt;/code&gt; is &lt;code dir=&quot;auto&quot;&gt;ratified&lt;/code&gt; in the index (with one broken
fixture per rule). It was never written in core.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The &lt;code dir=&quot;auto&quot;&gt;PROMISE_INDEX_SYNC_TOKEN&lt;/code&gt; org sync&lt;/strong&gt; — the org-owned sync secret of Open Question 1
above, which would have opened the index-update pull requests. It is not created or scoped
in core.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Turning a promise into a scenario&lt;/strong&gt; (the promise→scenario mapping) — the act of deriving
a Gherkin scenario from a ratified promise. That mapping lives with the promise pages in
spec-kitty-qa.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Because none of the four was ever built, this amendment records a &lt;strong&gt;narrowing of a
never-built plan&lt;/strong&gt;, not the deletion of code.&lt;/p&gt;
&lt;h3 id=&quot;consequences-of-the-narrowing&quot;&gt;Consequences of the narrowing&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The &lt;code dir=&quot;auto&quot;&gt;scenarios&lt;/code&gt; ci-router group now gates NO per-PR job.&lt;/strong&gt; The accepted Decision had the
&lt;code dir=&quot;auto&quot;&gt;scenarios&lt;/code&gt; group (&lt;code dir=&quot;auto&quot;&gt;tests/scenarios/**&lt;/code&gt;) gate the architectural job — and that architectural
job was precisely the dropped promise-tag check (item 2). With the check gone, the group
gates nothing; it is a deliberately-ungated filter group, like &lt;code dir=&quot;auto&quot;&gt;ci&lt;/code&gt;. The &lt;code dir=&quot;auto&quot;&gt;acceptance_jvm&lt;/code&gt;
group (&lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/**&lt;/code&gt;) still gates no per-PR job, unchanged.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Core keeps the scenarios, the Cucumber-JVM runner and the nightly.&lt;/strong&gt; Gherkin files under
&lt;code dir=&quot;auto&quot;&gt;tests/scenarios/&amp;#x3C;area&gt;/&lt;/code&gt;, the &lt;code dir=&quot;auto&quot;&gt;acceptance-jvm/&lt;/code&gt; Gradle subproject, the &lt;code dir=&quot;auto&quot;&gt;SpecKittyDriver&lt;/code&gt;
port and the nightly &lt;code dir=&quot;auto&quot;&gt;acceptance-bdd&lt;/code&gt; job all stay in core as the Decision describes.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tag semantics in core.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;@promise:&amp;#x3C;id&gt;&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;@defect:&amp;#x3C;issue&gt;&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;@qa:SC-A-NNN&lt;/code&gt; are
&lt;strong&gt;inert in core&lt;/strong&gt; — there is no index to check them against and no per-PR check reads them.
Only &lt;code dir=&quot;auto&quot;&gt;@wip&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;@manual&lt;/code&gt; carry meaning: each excludes a scenario from the gating nightly
run (&lt;code dir=&quot;auto&quot;&gt;not @wip and not @manual&lt;/code&gt;). &lt;code dir=&quot;auto&quot;&gt;@manual&lt;/code&gt; is an allowed tag, reserved for heavy or
on-demand scenarios.&lt;/li&gt;
&lt;/ul&gt;</content>
</entry>
