Pull the safety cord: unclog a deadlocked Actions queue
Who is this for (1)
- Stakeholder Persona: Maintainer — Written for the Maintainer reader.
When to use this
Use this guide when GitHub Actions runs on spec-kitty/spec-kitty stay queued for a long time
and the PRs you are landing cannot get a green signal. On 2026-11-01 about 480 runs were queued
or in progress at the worst point: 391 pull_request, 78 workflow_run, and a few push and
schedule runs.
Check the shape of the queue first:
unset GITHUB_TOKEN # if gh reports missing scopesR=spec-kitty/spec-kittygh run list --repo $R --status queued --limit 500 \ --json databaseId,workflowName,event,headBranch \ | jq -r 'group_by(.workflowName)[] | "\(length)\t\(.[0].workflowName)"' | sort -rnRepeat with --status in_progress, and group by .event as well. A queue dominated by
pull_request runs across many branches is the case this guide covers.
Steps
The label only works once .github/workflows/pr-pause-ci.yml and the label guards are on main.
PR runs use the workflows as merged with current main, so paused PRs need no rebase.
-
Decide the keep-set. List the PRs you are actively landing. Everything else gets paused.
-
Test the label on one PR.
Terminal window gh pr edit <n> --repo $R --add-label pr:pause-cigh run list --repo $R --workflow pr-pause-ci.yml --limit 5Confirm that the
PR Pause CIrun succeeded and that the PR’s three required gates (router gate,CI Modules gate,Clean install verification) readfail. -
Label every other open PR.
Terminal window gh pr list --repo $R --state open --limit 500 --json number,labels \| jq -r '.[] | select(all(.labels[]; .name != "pr:pause-ci")) | .number' \| grep -vxF -f keep-set.txt \| xargs -P 8 -I{} gh pr edit {} --repo $R --add-label pr:pause-cikeep-set.txtholds one PR number per line. -
Drain the queue if the pause runs are stuck behind it. Cancel every queued or in-progress run except
PR Pause CI, and repeat about every 20 seconds until the pause runs complete:Terminal window for s in queued in_progress; dogh run list --repo $R --status $s --limit 500 --json databaseId,workflowName \| jq -r '.[] | select(.workflowName != "PR Pause CI") | .databaseId'done | xargs -P 8 -I{} gh run cancel {} --repo $RExpect
CI Aggregateruns to reappear as you cancelCI Modulesruns (see below). -
Re-run the keep-set’s canceled runs. For each keep-set PR, find the canceled
pull_requestruns on its head SHA and re-run them:Terminal window SHA=$(gh pr view <n> --repo $R --json headRefOid -q .headRefOid)gh run list --repo $R --commit $SHA --status cancelled --json databaseId \-q '.[].databaseId' | xargs -I{} gh run rerun {} --repo $R -
Resume a paused PR when you pick it up for landing. Remove the label. The required workflows start a full run on the PR’s current head; no push is needed. If the gates still read “CI is paused” with the label off (a run started by adding the label was created after the one started by removing it, and cancelled it), add and remove the label again.
Terminal window gh pr edit <n> --repo $R --remove-label pr:pause-ci
Verify gh flags against gh <command> --help before running a loop.
On 2026-11-01 this took the in-flight count from 195 to 35, all from the keep-set, with 83 PRs paused. A fleet push to a paused PR now costs three quick gate jobs.
Why each step works
- Why the queue clogs. An external contributor fleet rebases and force-pushes dozens of open
PRs every time
mainmoves. Each push queues the full PR matrix: CI Router, CI Modules (about 60 jobs), CI Quality, Packs, Contracts, ci-windows, Release Readiness and more. A PR that lands onmainstarts a new wave. - Why the label stops it. Under
pr:pause-cithe root job of each PR workflow skips, so the heavy jobs skip. The three required gates still run and fail with a message, so the PR cannot be merged on GitHub while paused. Onlabeled, a smallpull_request_targetworkflow cancels the PR’s queued and in-progress runs and posts failing gate checks. The three required workflows trigger on the label change themselves: removing the label starts their full run, and adding it re-reports paused. Onunlabeledthe pause workflow re-runs the other runs it canceled, unless the PR’s head moved since (a push already started fresh runs). Details: Pausing a PR’s CI. - Queued PRs are not paused by the label. A merge-queue run carries no PR labels, so
pr:pause-cidoes not stop it. To stop a queued PR, remove it from the merge queue; a paused PR cannot enter the queue, because its required gates readfail. - Why test on one PR first. A broken guard on 80 PRs is 80 PRs to repair.
- Why drain. The pause runs wait in the same queue as everything else. Cancelling the backlog lets them start.
- Why the drain refills.
ci-aggregate.ymlis triggered byworkflow_runon CI Modules completion, and a canceled run counts as completed. Mass-canceling CI Modules queues one CI Aggregate run each (90 of 195 in-flight runs at one point). A single cancel sweep refilled within minutes, so loop until the pause runs finish. - Why no push is needed after unlabeling. Re-running a run that started while the PR was
labeled replays the label and pauses again, and the pause workflow cannot push for you: events
caused by
GITHUB_TOKENstart no workflow runs. So the required workflows trigger onunlabeledthemselves (#7019). A label other thanpr:pause-cistarts a no-op run of them that skips every job and cancels nothing.
What not to do
- Do not mark PRs draft. No workflow checks draft status, and undrafting fires
ready_for_review, which queues another full run. - Do not add a new skip label in the same crisis. The change that teaches workflows to honor it
must itself pass the clogged CI. (The
pr:pause-cichange, PR #6961, was merged on GitHub by operator override because its own CI could not get a runner.) - Do not rely on a cancel sweep alone. The fleet and CI Aggregate refill the queue.
- Shell pitfalls. In zsh,
"$BR:refs/..."eats:r; write"${BR}:refs/...". Neverpkill -fa pattern that appears in your own command line.
Residuals
- The root cause, the fleet’s rebase on every move of
main, is outside our control. ci-aggregate.ymldoes not skip canceled or paused sources; a candidate follow-up.scripts/ci/fleet_verdict.pyreads a paused PR as red.mission-status-api.ymlis not guarded by the label.