<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xml:base="https://docs.spec-kitty.ai/adr/3.x/2026-04-20-1-mutation-testing-as-local-only-quality-gate/">
  <id>https://docs.spec-kitty.ai/adr/3.x/2026-04-20-1-mutation-testing-as-local-only-quality-gate/</id>
  <title>Mutation Testing as a Local-Only Quality Gate</title>
  <updated>2026-04-20T00:00:00.000Z</updated>
  <author><name>Spec Kitty</name></author>
  <link rel="alternate" type="text/html" href="https://docs.spec-kitty.ai/adr/3.x/2026-04-20-1-mutation-testing-as-local-only-quality-gate/"/>
  <summary>Adopts mutmut as a local-only developer quality gate over the source tree, with the methodology captured as doctrine and CI integration deliberately deferred.</summary>
  <content type="html">&lt;h2 id=&quot;context-and-problem-statement&quot;&gt;Context and Problem Statement&lt;/h2&gt;
&lt;p&gt;Line and branch coverage measure &lt;em&gt;whether&lt;/em&gt; tests execute code; they do not measure whether tests would
detect real bugs. Several Spec Kitty modules have had high coverage but weak assertions — tests that
run the code path and assert only that nothing raised. When we wanted an objective signal of test
&lt;strong&gt;effectiveness&lt;/strong&gt; (not just reach), we needed a different tool.&lt;/p&gt;
&lt;p&gt;The question was not &lt;em&gt;whether&lt;/em&gt; to adopt mutation testing — the value is well-established — but at what
level of overhead, and with what scope.&lt;/p&gt;
&lt;h2 id=&quot;decision&quot;&gt;Decision&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Adopt &lt;code dir=&quot;auto&quot;&gt;mutmut&lt;/code&gt; 3.5.0 as a local-only developer quality gate.&lt;/strong&gt; Capture the methodology as
first-class doctrine. Defer CI integration until the workflow is proven and the contributor-facing
guidance is written.&lt;/p&gt;
&lt;p&gt;Concretely:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Tooling:&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;mutmut&lt;/code&gt; added under &lt;code dir=&quot;auto&quot;&gt;[project.optional-dependencies.test]&lt;/code&gt;. Configuration lives in
&lt;code dir=&quot;auto&quot;&gt;[tool.mutmut]&lt;/code&gt; in &lt;code dir=&quot;auto&quot;&gt;pyproject.toml&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Scope:&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;paths_to_mutate = [&quot;src/&quot;]&lt;/code&gt; — the entire first-party source tree is in scope when a
developer invokes &lt;code dir=&quot;auto&quot;&gt;mutmut run&lt;/code&gt; locally.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Doctrine:&lt;/strong&gt; the curated artifact set landed in
&lt;code dir=&quot;auto&quot;&gt;src/doctrine/tactics/built-in/testing/mutation-testing-workflow.tactic.yaml&lt;/code&gt;,
&lt;code dir=&quot;auto&quot;&gt;src/doctrine/styleguides/shipped/mutation-aware-test-design.styleguide.yaml&lt;/code&gt;, and language-specific
toolguides for Python (&lt;code dir=&quot;auto&quot;&gt;mutmut&lt;/code&gt;) and TypeScript (&lt;code dir=&quot;auto&quot;&gt;stryker&lt;/code&gt;). The DRG graph anchors these to
&lt;code dir=&quot;auto&quot;&gt;DIRECTIVE_034&lt;/code&gt; (Test-First Development).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No CI gate. No release-blocking check. No dashboard.&lt;/strong&gt; Mutation score is something a developer
checks before handing a mission over for review — a local-only signal, not a shared scoreboard.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;rationale&quot;&gt;Rationale&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Mutation runs are slow and flaky in constrained environments.&lt;/strong&gt; A full &lt;code dir=&quot;auto&quot;&gt;mutmut run&lt;/code&gt; against spec-
kitty takes hours and needs a roomy sandbox. Putting this behind a CI gate would make CI roughly 20×
slower and would fight with the &lt;code dir=&quot;auto&quot;&gt;mutmut 3.5.0&lt;/code&gt; trampoline bug that breaks any test spawning
&lt;code dir=&quot;auto&quot;&gt;python -m specify_cli&lt;/code&gt; as a subprocess.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Doctrine beats tooling when you haven’t adopted the discipline.&lt;/strong&gt; The kill-the-survivor workflow
requires judgement about which mutants are equivalent vs. which signal weak assertions. Without the
&lt;code dir=&quot;auto&quot;&gt;mutation-aware-test-design&lt;/code&gt; patterns (Boundary Pair, Non-Identity Inputs, Bi-Directional Logic) in
muscle memory, a CI gate just teaches contributors to game the score. Ship the doctrine, prove the
workflow locally, &lt;em&gt;then&lt;/em&gt; gate.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Signal-to-noise is better at the developer’s desk.&lt;/strong&gt; Contributors triaging their own surviving
mutants learn the patterns; contributors triaging another team’s failing CI run learn to mark
mutants “equivalent” and move on.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;current-state-as-is&quot;&gt;Current state (As-Is)&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Each mutmut invocation forks a &lt;code dir=&quot;auto&quot;&gt;mutants/&lt;/code&gt; sandbox directory and runs the project’s test suite against
each injected mutant. Baseline-test failures in the sandbox currently prevent &lt;code dir=&quot;auto&quot;&gt;mutmut run&lt;/code&gt; from
reaching the mutant-testing phase.&lt;/li&gt;
&lt;li&gt;We keep the sandbox baseline green via an explicit &lt;code dir=&quot;auto&quot;&gt;--ignore=&lt;/code&gt; list in
&lt;code dir=&quot;auto&quot;&gt;pyproject.toml[tool.mutmut].pytest_add_cli_args&lt;/code&gt;. Each entry targets tests that are incompatible
with the sandbox for one of four reasons:
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Subprocess CLI invocation:&lt;/strong&gt; tests that shell out to &lt;code dir=&quot;auto&quot;&gt;python -m specify_cli&lt;/code&gt; trip the known
&lt;code dir=&quot;auto&quot;&gt;mutmut 3.5.0&lt;/code&gt; trampoline bug (&lt;code dir=&quot;auto&quot;&gt;AttributeError: &apos;NoneType&apos; object has no attribute &apos;max_stack_depth&apos;&lt;/code&gt; in &lt;code dir=&quot;auto&quot;&gt;record_trampoline_hit&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Whole-codebase AST walks:&lt;/strong&gt; pytestarch, chokepoint coverage, ownership invariant checks — these
blow the 30-second per-test timeout.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Wheel builds / multiprocessing spawn:&lt;/strong&gt; adversarial distribution tests and concurrency tests
that conflict with mutmut’s &lt;code dir=&quot;auto&quot;&gt;set_start_method(&apos;fork&apos;)&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Repo-state dependencies:&lt;/strong&gt; tests that read files outside &lt;code dir=&quot;auto&quot;&gt;also_copy&lt;/code&gt;d directories (live
&lt;code dir=&quot;auto&quot;&gt;kitty-specs/&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;scripts/&lt;/code&gt;, cross-mission fixtures).&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Additionally, marker-based deselection removes &lt;code dir=&quot;auto&quot;&gt;slow&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;e2e&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;distribution&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;adversarial&lt;/code&gt;,
&lt;code dir=&quot;auto&quot;&gt;windows_ci&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;platform_darwin&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;live_adapter&lt;/code&gt;, and &lt;code dir=&quot;auto&quot;&gt;architectural&lt;/code&gt; tests.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;type_check_command&lt;/code&gt; is disabled because mutmut’s line-based JSON parser cannot handle mypy’s
multi-line &lt;code dir=&quot;auto&quot;&gt;hint&lt;/code&gt; fields. The cost is accepting that mutants producing type errors will be tested
rather than filtered out; they typically get killed on import failure.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The &lt;code dir=&quot;auto&quot;&gt;--ignore=&lt;/code&gt; list is a brittle external registry — it goes stale when tests move or rename, and the
reason a test is excluded lives in a TOML comment rather than next to the test itself.&lt;/p&gt;
&lt;h2 id=&quot;target-state-landed-2026-04-20&quot;&gt;Target state (landed 2026-04-20)&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;The explicit per-file &lt;code dir=&quot;auto&quot;&gt;--ignore=&lt;/code&gt; list has been replaced with two pytest markers:
&lt;code dir=&quot;auto&quot;&gt;non_sandbox&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt;.&lt;/strong&gt; The two categories capture different exclusion intents and must
not be conflated.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Register both markers&lt;/strong&gt; in &lt;code dir=&quot;auto&quot;&gt;[tool.pytest.ini_options].markers&lt;/code&gt;:
&lt;ul&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;non_sandbox: test is structurally incompatible with mutmut&apos;s forked sandbox — subprocess CLI calls, whole-codebase AST walks, wheel builds, or repo-state fixtures that fall outside mutmut&apos;s also_copy tree. Exclusion is deterministic: the test will never pass in the sandbox until the underlying sandbox constraint changes.&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;flaky: test passes reliably in the main test suite but produces non-deterministic results under mutmut (or other mutation/forking pipelines) — e.g., race conditions in target-branch detection, time-sensitive assertions, implicit reliance on process-wide state. Re-evaluate on every pipeline tooling upgrade; aim to fix root causes and remove the marker.&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;At the top of each currently-ignored test file, add &lt;code dir=&quot;auto&quot;&gt;pytestmark = pytest.mark.&amp;#x3C;marker&gt;&lt;/code&gt; with a
one-line comment stating &lt;em&gt;why&lt;/em&gt;. For wholesale-ignored directories, use a directory-level
&lt;code dir=&quot;auto&quot;&gt;conftest.py&lt;/code&gt; that applies the marker collectively.&lt;/li&gt;
&lt;li&gt;Collapse the long &lt;code dir=&quot;auto&quot;&gt;--ignore=&lt;/code&gt; block in &lt;code dir=&quot;auto&quot;&gt;pyproject.toml&lt;/code&gt; into two deselection clauses:
&lt;code dir=&quot;auto&quot;&gt;&quot;-m&quot;, &quot;... and not non_sandbox and not flaky&quot;&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Why the distinction matters:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;non_sandbox&lt;/code&gt; tests are &lt;strong&gt;accepted losses&lt;/strong&gt; — we’ve decided the coverage trade-off is worth the
stability. No follow-up is planned unless the upstream trampoline bug is fixed.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; tests are &lt;strong&gt;debt&lt;/strong&gt; — they indicate real test-suite weaknesses (hidden global state,
timing dependencies, missing fixture isolation). Each &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; marker should have a TODO comment
pointing to the root cause; they should shrink over time, not accumulate.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Benefits:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Co-located intent.&lt;/strong&gt; The “why” sits next to the test, not in a separate config file.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rename-safe.&lt;/strong&gt; Moving or renaming a test no longer silently removes it from the ignore list.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Individually re-enableable.&lt;/strong&gt; When the upstream mutmut trampoline bug is fixed, we remove the
&lt;code dir=&quot;auto&quot;&gt;non_sandbox&lt;/code&gt; marker from subprocess tests without touching the TOML.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Discoverable.&lt;/strong&gt; &lt;code dir=&quot;auto&quot;&gt;pytest -m non_sandbox --collect-only&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;pytest -m flaky --collect-only&lt;/code&gt;
list exactly which tests are excluded and for which reason — replacing &lt;code dir=&quot;auto&quot;&gt;grep&lt;/code&gt; archaeology on
&lt;code dir=&quot;auto&quot;&gt;pyproject.toml&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Signals tech debt.&lt;/strong&gt; The &lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; bucket surfaces tests that may be lying to us in CI; the
&lt;code dir=&quot;auto&quot;&gt;non_sandbox&lt;/code&gt; bucket does not.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;required-follow-up-work&quot;&gt;Required follow-up work&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Contributor guidance document.&lt;/strong&gt; Landed 2026-04-20 at &lt;code dir=&quot;auto&quot;&gt;docs/how-to/run-mutation-tests.md&lt;/code&gt;.
Covers running &lt;code dir=&quot;auto&quot;&gt;mutmut run&lt;/code&gt;, reading results, the kill-the-survivor workflow, equivalent-mutant
suppression, and how to add the &lt;code dir=&quot;auto&quot;&gt;non_sandbox&lt;/code&gt;/&lt;code dir=&quot;auto&quot;&gt;flaky&lt;/code&gt; markers to new tests that won’t run in
the mutant sandbox.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;User-facing rationale.&lt;/strong&gt; A short section in the main &lt;code dir=&quot;auto&quot;&gt;README.md&lt;/code&gt; or a dedicated explanation doc
describing &lt;em&gt;why&lt;/em&gt; mutation score matters and how it differs from coverage — enough to prevent a
contributor from treating the two as interchangeable.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Re-evaluate CI integration&lt;/strong&gt; once (a) the &lt;code dir=&quot;auto&quot;&gt;mutmut 3.5.0&lt;/code&gt; trampoline bug is fixed upstream or
worked around at the project level, and (b) the doctrine is known to be internalised — likely
measured by contributors citing the &lt;code dir=&quot;auto&quot;&gt;mutation-aware-test-design&lt;/code&gt; patterns in code review.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;consequences&quot;&gt;Consequences&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Positive:&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Developers have a real tool to answer “if I introduced a bug here, would my tests fail?” rather
than the proxy question “did my tests run this line?”.&lt;/li&gt;
&lt;li&gt;Doctrine artifacts travel with the project: any contributor who picks up a mission inherits the
mutation-aware test design patterns.&lt;/li&gt;
&lt;li&gt;No CI slowdown. No review-board arguments about mutation score thresholds.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Negative:&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The &lt;code dir=&quot;auto&quot;&gt;--ignore=&lt;/code&gt; list will drift until the &lt;code dir=&quot;auto&quot;&gt;non_sandbox&lt;/code&gt; marker migration lands.&lt;/li&gt;
&lt;li&gt;Contributors who never run &lt;code dir=&quot;auto&quot;&gt;mutmut run&lt;/code&gt; locally get no mutation signal at all. Without CI this is a
soft policy, not an enforced gate.&lt;/li&gt;
&lt;li&gt;Mutmut’s subprocess-unfriendly trampoline forces ignores around spec-kitty’s CLI integration tests.
That’s a real coverage blind spot for CLI behaviour, mitigated (not eliminated) by the unit tests
that cover the same code paths in-process.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;related&quot;&gt;Related&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Tactic: &lt;code dir=&quot;auto&quot;&gt;src/doctrine/tactics/built-in/testing/mutation-testing-workflow.tactic.yaml&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Styleguide: &lt;code dir=&quot;auto&quot;&gt;src/doctrine/styleguides/shipped/mutation-aware-test-design.styleguide.yaml&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Toolguide: &lt;code dir=&quot;auto&quot;&gt;src/doctrine/toolguides/shipped/PYTHON_MUTATION_TOOLS.md&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Toolguide: &lt;code dir=&quot;auto&quot;&gt;src/doctrine/toolguides/shipped/TYPESCRIPT_MUTATION_TOOLS.md&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Import candidate: &lt;code dir=&quot;auto&quot;&gt;src/doctrine/_reference/craft-guidelines-library/candidates/mutation-testing-skillpack.import.yaml&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Directive: &lt;code dir=&quot;auto&quot;&gt;DIRECTIVE_034&lt;/code&gt; (Test-First Development)&lt;/li&gt;
&lt;/ul&gt;</content>
</entry>
