Skip to content

CLI Acceptance Scenarios (Cucumber-JVM nightly)

Who is this for (1)

The acceptance scenarios check what the spec-kitty CLI promises its users, from the outside. They run the CLI built from the commit as a separate process and look only at what a user would see: files, exit codes and git state. A broken promise turns the nightly red on every supported operating system.

PartLocation
Scenarios (Gherkin)tests/scenarios/<area>/*.feature
Runner, step definitions, driveracceptance-jvm/ (Gradle, Cucumber-JVM 7, Java 25)
Nightly jobacceptance-bdd in .github/workflows/ci-nightly.yml
Decision recordsADR 2026-10-11-3 (acceptance testing and BDD support), ADR 2026-10-13-1 (promise links, per-PR check, evidence), ADR 2026-10-15-1 (ownership and absorption)
Absorbed engineering teststests/scenarios/absorption-ledger.json

Who owns what

PartOwnersReview
Scenarios, the absorption ledger, this guideQA (spec-kitty-qa) and corea scenario change asks a QA owner and a core owner (.github/CODEOWNERS)
Runner, step definitions, driver, fixture builders (acceptance-jvm/)corecore
Promise pages, promise index, trace lint, Allure reportspec-kitty-qaspec-kitty-qa

QA drafts a scenario from a promise and lands it as @wip; an engineer binds its steps and adds a positive control; @wip comes off in a pull request a QA owner and a core owner reviewed. tests/architectural/test_scenario_contract.py checks the CODEOWNERS rules on every pull request. Requiring code-owner review on GitHub is a repository setting, not set today.

Write a scenario

  1. Pick a promise the CLI makes to its users, such as a sentence from a command’s --help text. Name it in the scenario file’s maintainer notes.
  2. Write the scenario from the user’s point of view: what they have, what they run, what they get. Leave test mechanics out of the Feature and Scenario text.
  3. Order the steps Arrange, Assume, Act, Assert (see below), and reuse existing steps before adding new ones.
  4. Give the scenario a same-fixture positive control, so a command that did nothing cannot pass it. For example, “the folder .claude/skills exists” proves that init actually ran before “the file .kittify/config.yaml is unchanged” can mean anything.

tests/scenarios/README.md lists the available steps.

Step structure: Arrange, Assume, Act, Assert

Each lifecycle group has its own step class in the kitty.acceptance.steps package of the acceptance-jvm test sources. They share one ScenarioContext per scenario, injected by Cucumber’s PicoContainer integration.

GroupGherkinClassDoes
ArrangeGivenArrangeStepsBuilds the starting conditions and fails fast when that did not work.
AssumeGiven / AndAssumeStepsChecks that the starting state is what the scenario expects, and records it for later comparison.
ActWhenActStepsRuns the command under test.
AssertThen / AndAssertStepsChecks the post-conditions.

Steps talk to the CLI only through the SpecKittyDriver port and never import Python code. Assertions use AssertJ.

The one Arrange write that is not the CLI’s

ADR 2026-10-15-1 decision 6 lists what an Arrange builder may write: the planning files an agent authors, the code a work package’s implementer writes, a reviewer’s feedback file outside the repository, and plain files a user edits in git. It never writes CLI-managed state.

There is one narrow, documented exception, for a promise whose subject is the state the CLI reduces: a scenario that checks how the status log’s ordering is honoured may plant a row as a teammate’s git pull delivers it. MissionLifecycle.plantFutureStampedApproval restamps a work package’s approved row to a fixed ten minutes into the future and commits it, so the reducer sees a clock that ran ahead of the rejecting reviewer’s. The row keeps exactly the shape the CLI writes (no lamport_clock on lane transitions), and everything after it goes through the CLI. tests/scenarios/status/event-traceability.feature is the only user, added for #6260; do not copy the pattern for a behaviour a real CLI call can arrange.

Builders name the topology they mean

Since #7282 (ADR 2026-11-06-3), agent mission create without --topology records an unsettled single_branch Mission: it switches the checkout onto a Mission topic branch and makes no coordination branch and no lane worktree. A builder that means a coordination Mission therefore asks for one: MissionLifecycle.planMission and createMission, and MissionFixture.createMission, pass --topology lanes_with_coord (MissionLifecycle.COORDINATION_TOPOLOGY), and planSingleBranchMission passes --topology single_branch. A scenario about the size default itself must call mission create with no --topology explicitly, and must commit or discard each Mission’s spec.md scaffold before the next create: a default create refuses a dirty checkout with WRITE_CHECKOUT_DIRTY, by design, so one Mission’s untracked scaffold never follows the checkout onto another Mission’s branch. A Python acceptance helper that pins SPEC_KITTY_HOME puts the home beside the project checkout, never inside it: cache files written there would dirty the write checkout, and the Mission-branch switch refuses a dirty one (#7342).

Tags

TagRule
@promise:<id>Names the promise the scenario checks. <id> matches [A-Za-z0-9][A-Za-z0-9._-]*, for example PR-CLI-INIT-001. Not repeated on one scenario.
@wipDraft or under repair. Needs @wip-owner:<handle> and @wip-since:<YYYY-MM-DD>. Runs in the separate non-gating acceptance-bdd-wip nightly job, which never fails a leg and nothing waits on (#7492).
@wip-owner:<handle>Owner of the draft. <handle> matches [A-Za-z0-9][A-Za-z0-9._-]*.
@wip-since:<YYYY-MM-DD>Date the scenario was marked @wip.
@manualNever runs automatically.
@defect:<issue>Links a known defect. <issue> is a positive integer.
@qa:<ref>Reference for the QA repository. <ref> uses the identifier character set.
@<bare>Bare tag matching [A-Za-z0-9][A-Za-z0-9_-]*, with no :.
noneRuns in the gating nightly. A failure fails that OS leg.

Any other tag containing : is an error. Tags on a Feature or Rule apply to every scenario under it, and tags on an Examples: block apply to the rows of that table. Everything except @wip and @manual gates the nightly.

Linking a scenario to a promise

A scenario links to a promise by identifier only, with @promise:<id>. The scenario carries no promise wording, rationale or status. This repository checks the shape of the identifier and nothing more; resolving it to a promise happens in the QA repository.

Drafted scenarios

A scenario drafted from a promise page lands as @wip, with its @promise:<id> tag, @wip-owner and @wip-since. It may use steps that have no binding yet. An engineer then binds the steps, adds a positive control and removes @wip. Tooling never invents an identifier.

Removing @wip and adding @promise:<id> are separate steps. The engineer’s pull request binds the steps, adds the positive control and drops @wip; the identifier comes from the QA promise registry, so if it is not assigned yet the scenario gates without it (the contract check validates the identifier’s shape only when the tag is present) and a QA owner adds the tag in review. Never hold a bound scenario in @wip until an identifier arrives: an unbound promise that nobody executes is the worse outcome.

Per-PR check

tests/architectural/test_scenario_contract.py validates the feature files, the tags and the step bindings on every pull request. A feature file with no scenario fails it with NO_SCENARIOS, so a misspelt Scenario: keyword cannot hide its steps as description text. Run it locally with:

Terminal window
uv run --frozen pytest -q tests/architectural/test_scenario_contract.py

The gating run has no retries. A flaky scenario is a defect to fix, not to retry (see the flakiness policy).

Absorb an engineering test

A test in tests/integration or tests/e2e that checks something a user can see from the built CLI can move into a scenario. The rules are in the test-pyramid rebalance plan; in short:

  1. Write an untagged scenario with at least the test’s assertions, on an equal or stronger arrange.
  2. Run it, then put the defect the test guards back in the source and watch the scenario go red. Restore the source.
  3. Delete the test in the same pull request, and add a row to tests/scenarios/absorption-ledger.json naming the retired test, the scenario that replaces it, the issue and a note on any difference.

The per-PR check fails when a retired test still exists, a replacement scenario is missing or is @wip/@manual, or a file the ledger’s baseline lists was deleted without a row. A red gating night on main files one P0 issue through the acceptance-bdd-escalation job, as a red integration-next night does.

Run the scenarios locally

You need JDK 25 and the pinned Gradle. Install Gradle with the shared installer, which checks the download against its pinned sha256:

Terminal window
python contracts/tools/install_tools.py --pins acceptance-jvm/tools/pins.json --dest "$HOME/.cache/spec-kitty-tools" --only gradle
export PATH="$HOME/.cache/spec-kitty-tools/gradle-8.14.5/bin:$PATH"
gradle -p acceptance-jvm test --no-daemon -Pacceptance.pass=gating

-Pacceptance.pass=gating runs not @wip and not @manual; -Pacceptance.pass=wip runs the @wip scenarios. The property selects the tag filter and the output folder together. -Pacceptance.shard=K/N runs shard K of N of the pass: the scenarios the tag filter would run, ordered by feature path and line and dealt out round-robin, so the shards are disjoint and together run every scenario. The nightly runs the gating pass in three shards on Windows, two on macOS and one on Linux.

The runner builds a wheel from your checkout with uv build and installs it into a throwaway virtual environment, so it tests your working tree as a user would install it. Each scenario gets its own temporary directory, home directory and git repository. Scenarios run four at a time (the runner’s junit-platform.properties), so a step must never share mutable state across scenarios. The test task never resolves UP-TO-DATE: the wheel is rebuilt on every run, so a change under src/ is always tested.

Each pass writes only under acceptance-jvm/build/acceptance/<pass>/, so running wip after gating leaves the gating results untouched:

  • JUnit XML: acceptance-jvm/build/acceptance/<pass>/junit/
  • Cucumber JSON: acceptance-jvm/build/acceptance/<pass>/cucumber.json
  • Allure results: acceptance-jvm/build/acceptance/<pass>/allure-results/

The nightly’s evidence script also writes acceptance-jvm/build/acceptance/evidence.json, counted from the JUnit XML of the passes. The nightly runs the two passes in separate jobs (acceptance-bdd for gating, acceptance-bdd-wip for @wip, #7492), so each job’s bundle lists the pass it ran in passes_run and reports the other pass as missing. It is also given the outcome of the step that ran each pass, so a run that stopped after writing some passing JUnit XML is recorded as failed. Only a gating pass that did not pass fails the evidence step.

A gating run fails if no scenario ran, so a mistyped tag filter or a wrong feature path cannot pass as green. Only the @wip run may find none.

Each scenario also checks its own isolation. The CLI must write something under the redirected home directory, which shows the environment redirection took effect. The real ~/.spec-kitty must be left untouched. Lock files are ignored in that comparison, because another spec-kitty process on your machine may touch them.

On a host that runs several Missions at once, export SPEC_KITTY_ACCEPTANCE_SHARED_HOST=1 (1, true, yes or on) before the run. The comparison then also skips the two runtime-state subtrees that any concurrent spec-kitty process writes: spec-kitty-prompts/ (prompt temp files, one directory per repository) and gate-locks/ (machine-wide cross-process gate locks). Every other part of the real root stays compared, including credentials and any new top-level file a redirection bug would leave behind. The switch is ignored on a GitHub Actions runner, so the nightly always keeps the whole-tree comparison, and each scenario runs a built-in control that proves a write outside the two skipped subtrees still fails it — the narrowing can never quietly become a check that cannot fail. kitty/acceptance/environment/RealHomeFingerprint.java is the single owner of that list; this page only points at it.

View the Allure report

acceptance-jvm/build/acceptance/gating/allure-results/ holds raw result JSON, not a report you can open in a browser. Turn it into the HTML report with the Allure command-line tool. If you have it installed, serve builds the report into a temporary directory and opens it:

Terminal window
allure serve acceptance-jvm/build/acceptance/gating/allure-results

Without a local install, run the same tool through npx (it needs a JDK on PATH):

Terminal window
npx allure-commandline serve acceptance-jvm/build/acceptance/gating/allure-results

To keep the report rather than serve it from a temporary directory, generate it into a folder and open that folder:

Terminal window
allure generate acceptance-jvm/build/acceptance/gating/allure-results --clean -o acceptance-jvm/allure-report
allure open acceptance-jvm/allure-report

The nightly does not build this report; it uploads the raw Allure results inside the acceptance-bdd-results-<os>-shard-<k> artifacts, one per shard (the @wip pass in acceptance-bdd-wip-results-<os>). To view a nightly run as a report, download that artifact and run one of these commands on its gating/allure-results/ (or wip/allure-results/) directory.

Change a dependency

acceptance-jvm/gradle/verification-metadata.xml pins every dependency by sha256. It is generated, never edited by hand. After changing acceptance-jvm/build.gradle, regenerate it and commit the result:

Terminal window
gradle -p acceptance-jvm --write-verification-metadata sha256 test --no-daemon

acceptance-jvm declares no Gradle version of its own and commits no wrapper; the nightly installs Gradle from acceptance-jvm/tools/pins.json. The nightly’s pin-guard step runs contracts/tools/gradle_pin_check.py, which for acceptance-jvm checks only that the url in pins.json carries the pinned version (and a gradle-version: input of the acceptance-bdd job, if one is ever added). It does not check the plugin or dependency pins; the sha256 verification-metadata.xml covers those.

Read a nightly failure

The acceptance-bdd job runs once per OS and shard. Open the failed leg, then download its acceptance-bdd-results-<os>-shard-<k> artifact. The Cucumber JSON and JUnit XML name the failed step and carry its assertion message, including the command’s output. The spec-kitty-qa nightly also collects these results.