ADR: Gate outcomes carry a typed severity; an operator-configured error-handling strategy decides the CLI effect
Gate outcomes carry a typed severity; an operator-configured strategy decides the CLI effect
Filename: 2026-08-13-6-gate-outcomes-carry-severity-operator-strategy-decides-effect.md
Status: Accepted (2026-09-27, amended in place; amended 2026-11-12)
Date: 2026-08-13 (proposed) · 2026-09-27 (amended and accepted) · 2026-11-12 (amended)
Deciders: Operator (ATDD)
Technical Story: Resolves the fail-closed (ADR 2026-08-13-2) vs skip-and-proceed (ADR 2026-08-13-4) contradiction the dialectics pass surfaced. Instead of a per-gate boolean disposition, gate outcomes are typed by severity and the CLI effect is an operator policy. Implementation tracks #3416 (minimal kernel ladder) and #3417 (strategy).
Amendment note (2026-09-27)
Accepted together with ADR -2 and ADR -4 as one set, after the pack-trust ADR review (architect-alphonso / paula-patterns squad, operator-approved 2026-09-27). The body is rewritten in place. Changes against the Proposed text:
- Contradiction fixed. The Proposed text defined
RECOVERABLEas “may proceed degraded” but made the defaultblock_above(RECOVERABLE), which blocks it. The default now blocks atBLOCKINGonly;RECOVERABLEproceeds degraded and is recorded loudly. - Could-not-run severities are fixed by the dispatcher, not declared by the gate. This answers
the 2026-08-15 fail-open reconcile question on #2535/#2599/#3417 with reading (a): machinery
faults are
RECOVERABLE, trust and tamper failures areBLOCKING. - Guardrail added: repository-committed configuration cannot demote trust or tamper outcomes
below
BLOCKING; the effective strategy is surfaced in every verdict. - Scoping (open question 4) resolved: a minimal kernel ladder ships first; consolidating the other Severity enums is follow-on debt (#3416), not a precursor.
- Stale reference fixed:
status/doctor.pySeverityis now at line 33.
Amendment (2026-11-12): a proven foreign evidence tree refuses move-task
Operator ruling 2026-11-12 (decision 01M7733YYK1615ERKAY41C2M7C, Mission
lane-code-verification-01M76AC0, #7554, #7547). Before this, a test run inside a lane
worktree could load the repository root checkout’s source through its editable install, so
the pre-review gate could pass a work package on code that was not the lane’s. The gate now
runs its tests through the lane’s own test invocation and probes, before it runs, which tree
that invocation imports. This amendment records how the gate treats the probe’s answer.
- New outcome
foreign_tree, a provenance refusal. When the probe proves the run would load code from another tree, the gate returnsforeign_treeand runs no test.move-task <WP> --to for_reviewrefuses withEVIDENCE_TREE_MISMATCHand exit 1, and the transition is not applied. This holds whateverreview.fail_on_pre_review_regressionsays: that setting governs new test failures, and a foreign tree is not a test failure but evidence that no lane test result exists. --forcebypasses it, on the record. A forced move proceeds and recordsforce_bypassed: trueandbypassed_refusal: "EVIDENCE_TREE_MISMATCH"; the outcome staysforeign_tree, because no test ran. This is the difference from the trust failures in the table below: pack trust, pin hash mismatch and tamper stayBLOCKINGwith no bypass. A foreign tree says where the code came from, not that executable doctrine is untrusted, so the operator may proceed knowingly.- Carve-out from “gate evaluation never breaks
move-task” (C-003). Only a proven foreign tree refuses. A probe that faults or times out, or a project the probe cannot inspect (for example one that is not a Python package), recordsunverifiedand the transition proceeds as before. Every other machinery fault keeps the C-003 promise and maps toRECOVERABLE. - Probed only where tests run. The probe runs, and the evidence tree is recorded in the
transition’s
pre_review_gate.evidence_tree(lane path, head, dirty flag, probe result), only when the gate is about to run the head’s tests: a declared test command or an override scope. The baseline run is not probed or recorded. A gate arm that runs no test (an undeclared gate staysskipped, an empty diff, no active binding) neither probes nor refuses. - The run environment changes for everyone. The head run (declared command and the
override tier’s interpreter fallback) and the baseline run no longer inherit
PYTHONPATH,VIRTUAL_ENVorUV_PROJECT_ENVIRONMENTfrom the caller whenever the gate builds a lane test invocation (every lane and write checkout; a planning-lane work package, or a build that failed open, keeps the inherited environment). A tree with a.venvof its own runs with that venv’s script directory first onPATH; a tree without one getsPYTHONPATH=<tree>/srcwhen it has trackedsrc/packages. The head run also gets the lane’s test database name; the baseline, which has no lane identity, does not. A project whose tests relied on an inheritedPYTHONPATHmust declare it in its test command. On the override tier’suv run --project <lane>path the run uses the environment uv resolves for the lane rather than the probed one, so there the evidence describes the lane’s invocation, not that exact process; the probe covers uv’s interpreter only when the lane has a.venvof its own. Whenuvis not on the run’sPATH(looked up on the lane’s environment, not the caller’s) or the lane has nouv.lock, the fallback runs the lane’s own.venvinterpreter when it has one, the interpreter the probe checked, and otherwise the CLI’s interpreter withPYTHONPATH=<tree>/src, which the probe also used. - What the probe covers. It checks the git-tracked
src/<package>/__init__.pypackages of the tree. A flat or namespace-package layout has nothing to check, so the result isunverifiedand the gate does not refuse. It reports where packages load from; it does not prove the code there is the lane’s reviewed code. The interpreter still starts normally, so.pthfiles andsitecustomizein its environment run. - The skip switch still skips everything. The pre-review gate’s skip flag and environment variable bypass the gate entirely, the probe and this refusal included. The amendment makes no stronger guarantee than that.
The outcome table gains one row (below). The default strategy and the no-demotion guardrail
are unchanged: foreign_tree is not read from the configured threshold at all.
Context and Problem Statement
The gate design produced two conflicting per-gate booleans: gates flip fail-closed (block on failure), but untrusted packs were to be skipped (proceed). Same un-run gate, opposite outcomes, decided by why it did not run — and the “skip” branch silently no-ops security gates in CI. A single boolean cannot express “proceed, but degraded, and record it,” which is the real operator need for many checks (a docs lint failing should not necessarily block a transition; a missing security gate should).
The live behaviour adds a constraint. The C-003 invariant — gate evaluation never breaks
move-task — is implemented by _mt_fail_open_gate
(src/specify_cli/cli/commands/agent/tasks_move_task.py, FR-013), which turns any gate
exception into an unverified NO_COVERAGE warning. Any new model must keep that promise for
machinery faults without extending it to security failures.
The codebase already has the ingredients, fragmented across five sites:
kernel.glossary_types.Severity (src/kernel/glossary_types.py:70), audit.models.Severity +
its threshold fail_on: Severity | None with severity <= fail_on_threshold
(src/specify_cli/audit/models.py:47,53,203), status.doctor.Severity
(src/specify_cli/status/doctor.py:33), the charter-lint SEVERITY_ORDER blocking ladder
(src/specify_cli/charter_runtime/lint/findings.py, consumed by analysis_report.py), and
the strictness literals (kernel.glossary_types.Strictness, runtime … schema.py). The
audit module is already exactly the “block above a threshold” pattern requested.
Decision Drivers
- One coherent model for “what happens when a check does not pass,” across failed / degraded / informational / could-not-run outcomes.
- Operator owns strictness, not each gate author — mirrors linter/typechecker strictness config.
- Security and trust failures fail closed; machinery faults never silently pass and never break
move-task(C-003). - Do not add a sixth Severity enum; converge on one canonical ladder — without making that convergence a blocker for the gate work.
Decision Outcome
A gate produces a typed outcome; the operator’s error-handling strategy maps it to a CLI effect.
-
Severity ladder (
ERROR_SEVERITY):BLOCKING>RECOVERABLE>WARN>INFO.RECOVERABLEmeans “the system proceeds in a degraded fashion”: the outcome is recorded and surfaced loudly, and the transition is not stopped under the default strategy. -
Every gate outcome carries a severity. Who sets it depends on the outcome:
Outcome Severity Set by Untrusted pack, hash mismatch against pin or consent, tamper detected at dispatch (ADR -4) BLOCKINGdispatcher (fixed) Runner crash, timeout, missing interpreter, unconfinable host (ADR -2 baseline) RECOVERABLEdispatcher (fixed) Pre-review gate: the probe proves the lane’s tests would load code from another tree ( foreign_tree, amendment 2026-11-12)refusal EVIDENCE_TREE_MISMATCH;--forcebypasses and is recordedpre-review gate (fixed) Pre-review gate: the probe faults, times out or cannot inspect the project ( unverified)recorded, never refuses pre-review gate (fixed) A detected violation (the check ran and failed) the gate’s declared severitygate Pass — — Fail-open therefore survives only for machinery faults, and only as a recorded degraded outcome, never as a silent pass. Trust and tamper failures fail closed. This keeps C-003 for the grandfathered
spec-kitty-pre-reviewcode gate: its machinery faults map toRECOVERABLE, somove-taskstill completes, degraded and on the record. -
The operator selects an error-handling strategy in
.kittifyconfig, e.g.block_above(threshold)— outcomes at or above the threshold block the transition; below it, the CLI proceeds (degraded) and records the finding. This is the existingauditfail_onthreshold pattern, generalised. -
A default strategy ships: block at
BLOCKINGonly.BLOCKINGstops a transition;RECOVERABLEproceeds degraded and is recorded loudly;WARNandINFOare recorded. Because trust failures are fixed atBLOCKING, security gates fail closed out of the box, and CI without operator trust seeding blocks loudly rather than silently no-ops. -
Guardrail: no demotion of trust failures from the repository.
.kittify/config.yamlis committed, so a strategy read from repository configuration can make things stricter but can never make trust / tamper outcomes non-blocking. Those outcomes block regardless of the configured threshold. -
Surface the effective strategy. Every verdict records the effective strategy and where it came from, so “why did this pass?” always has an answer.
-
Canonical severity, minimal first.
ERROR_SEVERITYis defined once in the kernel (the layering-correct home, importable by doctrine) as a minimal ladder for gates, and the gate schema references it. Migrating the other four Severity enums onto it is follow-on debt (#3416 rescoped), not a precursor that blocks the gate work (#3417, #3418).
Consequences
Positive
- One knob for strictness; the fail-closed/skip contradiction dissolves — the operator’s threshold decides, uniformly, except for trust failures, which always block.
- The long-standing fail-open question has one answer that both keeps
move-taskunbreakable for machinery faults and closes the silent-skip hole for security. RECOVERABLEexpresses “proceed degraded,” which no boolean could.- A minimal kernel ladder unblocks the gate work now and is the anchor the later consolidation converges on.
Negative
- Until the follow-on consolidation lands there are temporarily six Severity definitions; the kernel ladder must be the one every new consumer uses.
- A permissive operator strategy can still weaken non-trust gates (a violation declared
BLOCKINGcan be demoted by the configured threshold); the effective-strategy record makes that visible rather than preventing it.
Neutral
- The strategy grammar (
block_above(threshold)vs a per-severity action map) is a config-schema detail (open question).
Resolved and open questions
- Open: extend/rename
kernel.glossary_types.Severity, or mint a newERROR_SEVERITYnext to it and migrate later? (Either way it lives in the kernel.) - Open: strategy grammar: single
block_above(threshold), or a per-severity action map ({BLOCKING: block, RECOVERABLE: warn, WARN: warn, INFO: log})? - Resolved (2026-09-27): could-not-run severities are fixed by the dispatcher; a detected violation carries the gate’s declared severity.
- Resolved (2026-09-27): a minimal kernel ladder ships inside the gate work; full consolidation is follow-on.