Contracts

adversarial-evidence-contract.md

Contract: Adversarial Evidence

Every adversarial-squad point-cut in this mission records each contested finding's disposition in research.md (Adversarial evidence ledger) as one of:

  • accepted — finding folded into plan/spec/code (state what changed).
  • changed — finding partially adopted (state the delta).
  • deferred_with_rationale — not adopted now; record why + any follow-up ticket.

No contested finding may be silently dropped. Point-cuts for this mission: post-plan (brownfield), post-tasks (anti-laziness), pre-merge (final). Supply-chain adversarial pass: N/A (no security-impacting dependency decision — see research.md Supply-chain note).

ci-workflow-contract.md

Contract: Nightly Integration Lane, Escalation, and Release Gate

Nightly integration lane (ci-nightly.yml, FR-006) (REVISED post-squad)

  • New job runs pytest tests/integration tests/next (directory selection, NOT -m integration alone — 843/1015 are integration-marked but no nightly marker suite selects them). Both roots are covered because the dead integration_tests_next special tier declared both; wiring only integration would leave tests/next/** (24 files) dark.
  • Retire the dead integration_tests_next special tier in ci-module-registry.yml in the same change (mandatory — GAP-3; closes #4729 honestly).
  • if: always() run-all-regardless; exit captured into env; terminal fail-loud step surfaces red (existing #4212 pattern). PWHEADLESS=1 + per-worker HOME isolation as elsewhere.
  • Add the job to nightly-summary.needs or its red won't surface in the aggregator.
  • Measurement: capture_shard_timings.py --module KeyErrors on a non-row dir, so measure via pytest tests/integration tests/next --durations=0 (or add a --test-dirs override to the tool). Shard only if the monolith overruns the per-shard timeout (NFR-004).
  • Does NOT add a modules[] row (C-003 — no per-PR promotion; that's #5037).

Escalation helper (scripts/ci/nightly_escalation.py, FR-007)

CLI: nightly_escalation.py --suite-key <key> --conclusion <success|failure> [--repo <owner/repo>] [--run-url <url>]

  • failure → find open issues containing <!-- nightly-escalation-key: <key> -->; if one exists, comment/update it; else create a priority:P0 issue with that marker in the body + a link to the failing run. INV-5: at most one open per key.
  • success → if an open issue with the key exists, close it (suite recovered).
  • No token / API error → print a warning, exit 0 (degrade to fail-loud only), never echo the token (C-005).
  • Unit-tested in tests/ci/test_nightly_escalation.py with a mocked GitHub client: covers create, update-existing (dedup), close-on-green, and token-absent degrade.

Release nightly gate (scripts/ci/release_nightly_gate.py + release.yml, FR-008) (REVISED post-squad)

CLI: release_nightly_gate.py --tag <release_tag> [--dispatch] [--timeout <s>]

  • Resolve <release_tag> → release_sha. Query ci-nightly.yml runs for head_sha == release_sha.
  • If a success run exists → exit 0 (gate passes).
  • Else if --dispatch → workflow_dispatch ci-nightly.yml by the tag ref (ref=<release_tag> — workflow_dispatch cannot take a bare SHA; the tag points at the release SHA so the run's head_sha matches), input mode=full; poll to terminal; exit 0 only if it concluded success, else non-zero.
  • In-flight nightly for the SHA → wait for terminal result (don't race), then decide.
  • Token: the dispatch uses RELEASE_NIGHTLY_DISPATCH_TOKEN (PAT or GitHub App) — the default GITHUB_TOKEN does NOT trigger a nested run (recursion guard). The nightly-gate job declares permissions: actions: write. Operator prerequisite: create this secret (call out in the PR body).
  • release.yml: new nightly-gate job; build-release gains needs: [nightly-gate] (and thus publish-pypi transitively). Fail closed (no publish) on any non-success. Token/secret absent on a real release ⇒ failure, never a silent pass.
  • Unit-tested in tests/ci/test_release_nightly_gate.py with a mocked GitHub client: green-exists→pass, none→dispatch(by tag)→green→pass, none→dispatch→red→fail, in-flight→wait, token-absent→fail-closed.

guards-contract.md

Contract: Coverage-Honesty Architectural Guards (REVISED per F17 — green-at-baseline)

Two new tests/architectural/ guards + one green-preserving fix. All import-based (no test execution), built on specify_cli.ast_analysis.imports primitives with a full ast.walk (deferred/function-level imports count). Scoped for speed (NFR-002): scan limited to resolved test dirs + the src package walk (~1682 files parsed once, well under 5 s per F17 measurement).

Landing posture: GREEN-at-baseline, NOT red-first. The guards pass at a committed measured baseline (recording honest current debt) and fail only on a regression (a new dark package/root). Remediation SHRINKS the baseline. This means no red guard ever reaches main's always-on tests/architectural/ battery (dissolves GAP-2). See research D9/F17.

Test-dir resolver (shared, pinned)

Resolve a row's test dirs with module-tests.yml precedence: explicit test_dirs if present, else tests/{module}. Never _canonical_test_mirror. Every resolved dir must exist on disk.

foreign-coverage guard (test_foreign_coverage_guard.py, FR-001)

For every ModuleRow whose roots: resolve to importable src packages:

  • Dark-root baseline (shrink-only): enumerate declared roots that NO test in the row's resolved dirs imports (today: next→specify_cli.runtime). Compare against a committed baseline in .github/ci-foreign-coverage-baseline.json. FAIL only when a new dark root appears or a covered root regresses. Do NOT hard-fail today's baseline set.
  • Ratio ratchet (shrink-only): the row's own-root import ratio ≥ its committed baseline. Improvement ratchets up.
  • Exemptions (recorded): non-src-root rows (ci: scripts/ci/, .github/workflows/); self-declared aggregate/inventory rows (execution_context, core_misc, unit, specify_cli_runtime).

Passes at baseline today (records next's dark root + all rows' current ratios). FR-004 (agent_utils explicit dir) ratchets agent's ratio up.

src-reachability guard (test_src_reachability_guard.py, FR-002)

Enumerate importable packages by fs-walking __init__.py subpackages + loose top-level *.py modules under the 6 wheel roots. Exclude data-only packages = a package/dir with no non-__init__ .py module (F18: covers both schemas (no __init__.py) AND data packages that HAVE an __init__.py but only ship .yaml/.json, e.g. charter.activation.corpus, charter.activation.packs, specify_cli.skills.data). Two tiers — both shrink-only baselines (F18: no absolute zero — import-based static analysis structurally cannot see subprocess/CLI-tested modules like specify_cli.__main__/encoding, so a hard zero-floor would false-positive forever):

  • Truly-dark baseline (shrink-only, hard-fail on GROWTH): a package imported by NO test anywhere (in-matrix ∪ out_of_matrix ∪ nightly). Measured 23 today (F18). Seed as committed baseline; FAIL only when a NEW truly-dark package appears. This is the strongest honesty ratchet: no new untested-anywhere code can land.
  • In-matrix-dark baseline (shrink-only): packages imported only by out-of-matrix/nightly dirs. Measured 27 today (F18, not the ~50 estimate). Seed as baseline; FAIL only when it GROWS. FR-005/009 shrink it by {live_work, config, calibration, tasks_authoring, diagnostics, bootstrap}; the guard asserts those left the baseline.

live_work is already in-matrix-reachable (via tests/zeitgeist_client) — FR-005 mints its row for per-PR execution of its 149 tests, and removes it from the in-matrix-dark baseline too if listed.

phantom-mirror fix (test_gate_selection_authority.py, FR-003) — GREEN-PRESERVING

  • Change the tree derivation to default tests/{module} (exists) instead of _canonical_test_mirror (phantom tests/agent_utils).
  • Add INV-2: every derived module test dir EXISTS on disk. Apply strict self-reachability only to explicit-test_dirs rows; apply INV-2 + non-empty-routing (#4454) to all rows (the agent row routes tests/agent→[cli, execution_context], its own-root debt, not a phantom).
  • Must stay GREEN (C-002) — baseline is 22 passed (F17).

Sidecar baseline (.github/ci-foreign-coverage-baseline.json)

Committed, MEASURED (documented provenance header). Holds: per-row own-root ratios, the dark-root set, and the in-matrix-dark package set. All shrink-only. WP01 seeds it at current measured values (guards green at baseline); WP02 shrinks it.

Unit tests (tests/ci/test_coverage_guards.py)

Exercise the lib directly: import-under-package (incl. a deferred import), resolver precedence, subpackage enumeration (excludes data dirs, includes loose modules), ratio computation, baseline shrink/grow logic (grow → fail, shrink → pass, equal → pass).