Skip to content

Comprehensive Manual Test Plan

Who is this for (1)

Spec Kitty: Comprehensive Manual Test Plan

Date: 2026-03-01 Scope: Full ecosystem verification across all repositories Goal: Validate Beta/GA readiness for the Spec Kitty SaaS platform


Prerequisites

Accounts Required

  • Spec Kitty SaaS staging account (https://team.spec-kitty.ai)
  • Jira Cloud test project with API token
  • Linear test workspace with API key
  • GitHub test repository with configured webhook
  • GitLab test project with configured webhook
  • Slack test workspace with bot token and app installed
  • Polar merchant account (staging/sandbox)
  • Stripe test account (existing, for comparison)
  • Nango account (nango.dev) with staging environment and connector configs

Local Environment

  • Python 3.11+ with spec-kitty-cli v2.0.2 installed (pip install spec-kitty-cli)
  • Git configured with SSH access to Priivacy-ai org
  • SPEC_KITTY_ENABLE_SAAS_SYNC=1 environment variable set
  • spec-kitty auth login completed successfully
  • Local test repository initialized with spec-kitty init
  • Browser with DevTools available (Chrome/Firefox)
  • Slack desktop or web client open

Staging Infrastructure

  • spec-kitty-saas deployed to Fly.io staging
  • PostgreSQL database accessible
  • Redis instance running (for Channels + Celery)
  • Celery workers running
  • Nango SaaS staging environment accessible (nango.dev)
  • Mailgun sandbox domain configured

Section 1: CLI Core Functionality

1.1 Installation and Version

#StepExpected ResultPass/FailNotes
1.1.1Run spec-kitty --versionDisplays 2.0.2 or later
1.1.2Run spec-kitty doctor skills (a bare spec-kitty doctor runs the default checks — command-files, state-roots, ops — and exits 0 when healthy, 1 on any finding)Skills/command-surface diagnostic passes, no critical errors
1.1.3Run spec-kitty auth statusShows authenticated user, team slug, server URL

1.2 Mission Lifecycle (software-dev)

Create a fresh test feature and run through the full lifecycle:

#StepExpected ResultPass/FailNotes
1.2.1Run spec-kitty specify with a short feature descriptionCreates kitty-specs/<NNN>-<slug>/spec.md with user stories and acceptance criteria
1.2.2Verify meta.json createdContains mission: "software-dev", feature slug, creation date
1.2.3Run spec-kitty planCreates plan.md with architecture decisions, tech stack, data model
1.2.4Run spec-kitty tasks (outline + packages + finalize)Creates tasks.md and tasks/WP*.md files with frontmatter
1.2.5Verify dependency frontmatterEach WP has dependencies: [] field, no cycles detected
1.2.6Run spec-kitty implement WP01Creates or reuses .worktrees/<slug>-lane-a/, switches to the lane branch
1.2.7Make a code change and commit in the worktreeCommit succeeds, WP status moves to in_progress
1.2.8Run spec-kitty agent tasks move-task WP01 --to for_reviewWP status commits to main, status.events.jsonl updated
1.2.9Run spec-kitty review on WP01Review workflow starts, can approve or request changes
1.2.10Move WP01 to doneStatus event appended, status.json materialized

1.3 Status Model

#StepExpected ResultPass/FailNotes
1.3.1Run spec-kitty agent tasks statusKanban board renders with correct lanes and WP counts
1.3.2Verify status.events.jsonlEvents are append-only, sorted keys, valid JSON per line
1.3.3Verify status.jsonSnapshot matches event log (deterministic reducer)
1.3.4Test transition guards: try planned → done directlyRejected (must go through intermediate lanes)
1.3.5Test doing aliasResolves to in_progress, alias never persisted in events
1.3.6Test force flag on terminal lane exitdone → in_progress only works with --force

1.4 Local Status View

The bundled local web dashboard has been removed; local status is read from the CLI.

#StepExpected ResultPass/FailNotes
1.4.1Run spec-kitty agent tasks status --mission <slug>Lane board printed with WPs in correct lanes and correct status counts
1.4.2Run spec-kitty agent tasks status --mission <slug> --jsonValid JSON with the same lane assignments
1.4.3Run spec-kitty orchestrator-api mission-state --mission <slug>Mission state and WP lanes match the status board
1.4.4Save the --json outputKept for parity comparison with SaaS (Section 8)

1.5 Merge System (Feature 017)

#StepExpected ResultPass/FailNotes
1.5.1Run spec-kitty merge --dry-run on a multi-WP missionConflict forecast shows predicted conflicts, auto-resolvable flags
1.5.2Run spec-kitty merge --mission <slug>Preflight validates the lane manifest, lane worktrees are clean, and target branch exists
1.5.3Run spec-kitty merge --dry-run --jsonJSON shows mission branch, target branch, and computed lanes
1.5.4Complete a full mergeLane branches merge into the mission branch, then the mission branch merges into target, and lane worktrees are cleaned up
1.5.5Verify status file auto-resolutionstatus.events.jsonl merge conflicts auto-resolved (append-both)

Section 2: Authentication and Sync

2.1 CLI Authentication

#StepExpected ResultPass/FailNotes
2.1.1Run spec-kitty auth login with valid credentialsTokens stored in ~/.spec-kitty/credentials, success message
2.1.2Run spec-kitty auth login with invalid credentialsClear error message, no tokens stored
2.1.3Run spec-kitty auth statusShows username, team slug, server URL, token expiry
2.1.4Run spec-kitty auth logoutCredentials removed, confirmation displayed
2.1.5Attempt sync operation while logged outPrompts to log in first
2.1.6Wait for access token expiry (or simulate)Auto-refresh happens silently, next operation succeeds

2.2 Event Sync (CLI → SaaS)

#StepExpected ResultPass/FailNotes
2.2.1Create a feature while authenticatedFeatureCreated event emitted, visible in SaaS within 5 seconds
2.2.2Move a WP to in_progressWPStatusChanged event synced, SaaS dashboard updates
2.2.3Move a WP to for_reviewEvent synced, SaaS shows WP in review lane
2.2.4Move a WP to doneEvent synced with evidence, SaaS shows WP complete
2.2.5Disconnect network, make status changesEvents queued in ~/.spec-kitty/queue.jsonl
2.2.6Reconnect networkQueued events batch-synced, SaaS catches up
2.2.7Verify Lamport clock orderingEvents arrive with monotonically increasing clock values
2.2.8Verify event envelope fieldsEach event has: event_id (ULID), event_type, timestamp, project_uuid, project_slug, team_slug, node_id, lamport_clock, payload

2.3 WebSocket Connection

#StepExpected ResultPass/FailNotes
2.3.1Observe WebSocket connection on loginConnection established, ping/pong heartbeat active
2.3.2Kill server temporarilyClient enters RECONNECTING state with exponential backoff
2.3.3Restart serverClient reconnects automatically, resumes sync
2.3.4Verify background sync intervalQueue flushes every 5 minutes (configurable)

Section 3: SaaS Dashboard

3.1 Account and Team Management

#StepExpected ResultPass/FailNotes
3.1.1Sign up with emailEmail confirmation sent, account created
3.1.2Verify emailAccount activated, redirected to dashboard
3.1.3Sign in with Google OAuthSuccessful authentication, team auto-created
3.1.4Sign in with GitHub OAuthSuccessful authentication, team auto-created
3.1.5Create a teamTeam created with slug, user is admin
3.1.6Invite a team memberInvitation email sent with accept link
3.1.7Accept invitation (as invitee)Invitee joins team with member role
3.1.8Verify team isolationMembers only see their team’s projects/features
3.1.9Toggle team feature flag (via Waffle)Feature appears/disappears for team members

3.2 Project and Feature Views

#StepExpected ResultPass/FailNotes
3.2.1View project listAll synced projects visible with slug and UUID
3.2.2Click into a projectFeatures listed with status summary counts
3.2.3Click into a featureKanban board with WPs in correct lanes
3.2.4Click into a WPWP detail page with status history, artifact links
3.2.5Verify real-time updatesWhen CLI changes WP status, SaaS board updates without page refresh

3.3 Collaboration Panel

#StepExpected ResultPass/FailNotes
3.3.1Open collaboration panel for a featurePanel renders with session info, participant list
3.3.2Verify participant presenceActive users shown with heartbeat indicator
3.3.3View collaboration timelineEvents listed chronologically with actor attribution
3.3.4Post a commentComment appears in timeline, visible to other participants
3.3.5View concurrent driver warningsIf two agents work same WP, warning displays
3.3.6Verify empty statesEmpty panel shows helpful guidance, not blank

3.4 Decision Inbox

#StepExpected ResultPass/FailNotes
3.4.1Navigate to decision inboxList of pending decisions visible
3.4.2View decision detailShows question, context, authority scope, alternatives
3.4.3Answer a decisionDecision captured with actor, timestamp, rationale
3.4.4View decision audit trailFull history: who asked, who answered, when, what alternatives
3.4.5Verify decision-point timelineDecision lifecycle events visible in collaboration timeline

3.5 Dossier (Mission Artifacts)

#StepExpected ResultPass/FailNotes
3.5.1View dossier for a featureArtifact index with file names, types, content hashes
3.5.2Click into an artifactContent rendered (markdown → HTML)
3.5.3Verify artifact status trackingActive, superseded, missing states shown correctly
3.5.4Verify dossier snapshotParity summary matches local artifacts

3.6 Glossary Dashboard

#StepExpected ResultPass/FailNotes
3.6.1View glossary projectionSemantic health status (healthy/warning/blocked)
3.6.2View conflict listActive conflicts with severity (high/medium/low)
3.6.3View top ambiguous termsMost-conflicted terms listed
3.6.4View recent clarificationsResolved conflicts with resolution history

Section 4: Connector Flows (E2E)

4.1 Jira Integration

#StepExpected ResultPass/FailNotes
4.1.1Connect Jira via OAuth in SaaSConnectorBinding created, status: connected, health: healthy
4.1.2View Jira connector health in dashboardHealth status visible, last sync time shown
4.1.3Create a Jira issue in test projectIssue exists in Jira with title and description
4.1.4Ingest Jira issue into spec-kitty missionIssue payload normalized to canonical mission-input schema
4.1.5Start spec-kitty specify from ingested issueSpecification pre-populated from Jira issue content
4.1.6Move WP to in_progress in spec-kittyJira issue transitions to corresponding status
4.1.7Move WP to for_review in spec-kittyJira issue transitions, comment added with spec-kitty link
4.1.8Move WP to done in spec-kittyJira issue resolved/closed, final comment with summary
4.1.9Verify bidirectional: change status in JiraSpec-kitty reflects the Jira status change
4.1.10Verify idempotency: replay same webhookNo duplicate events, no status regression
4.1.11Verify error handling: revoke Jira tokenConnector health degrades to degraded, ops log records failure
4.1.12Rotate credentials via NangoToken refreshed, connector returns to healthy
4.1.13Disconnect Jira in SaaSBinding revoked, audit event emitted

4.2 Linear Integration

#StepExpected ResultPass/FailNotes
4.2.1Connect Linear via OAuth in SaaSConnectorBinding created, healthy
4.2.2View Linear connector healthHealth visible in dashboard
4.2.3Create a Linear issueIssue exists in Linear
4.2.4Ingest Linear issue into spec-kittyCanonical mission-input created
4.2.5Start mission from Linear issueSpecification pre-populated
4.2.6Move WP through lifecycle in spec-kittyLinear issue status updates at each transition
4.2.7Verify bidirectional: change status in LinearSpec-kitty reflects change
4.2.8Verify idempotency on duplicate webhookNo duplicates
4.2.9Verify error → recovery cycleDegrade → fix → healthy
4.2.10Disconnect LinearBinding revoked, audit emitted

4.3 GitHub Issues Integration

#StepExpected ResultPass/FailNotes
4.3.1Connect GitHub via OAuthConnectorBinding created, webhook registered
4.3.2View GitHub connector healthHealth visible, webhook active
4.3.3Create a GitHub issue in test repoIssue exists on GitHub
4.3.4Ingest GitHub issue into spec-kittyCanonical mission-input created
4.3.5Start mission from GitHub issueSpecification pre-populated
4.3.6Move WP through lifecycleGitHub issue labels/status update
4.3.7Verify bidirectional: close issue on GitHubSpec-kitty reflects closure
4.3.8Verify webhook signature validationInvalid signature rejected with 403
4.3.9Verify webhook deduplicationSame delivery ID processed once
4.3.10Disconnect GitHubWebhook removed, binding revoked

4.4 GitLab Issues Integration

#StepExpected ResultPass/FailNotes
4.4.1Connect GitLab via OAuthConnectorBinding created, webhook registered
4.4.2Create GitLab issue and ingestCanonical mission-input created
4.4.3Start mission from GitLab issueSpecification pre-populated
4.4.4Move WP through lifecycleGitLab issue status updates bidirectionally
4.4.5Verify webhook signature validationInvalid token rejected
4.4.6Disconnect GitLabWebhook removed, binding revoked

4.5 Slack Integration

#StepExpected ResultPass/FailNotes
4.5.1Connect Slack via OAuth in SaaSBot token stored, binding created
4.5.2View Slack connector healthHealth visible in dashboard
4.5.3Verify bot appears in Slack workspaceBot user visible, can be invited to channels
4.5.4Trigger a stand-up notification (see Section 5)Message posted to designated channel
4.5.5Verify Slack → SaaS message relayResponses in Slack thread captured as decision events
4.5.6Disconnect SlackBot revoked, binding cleaned up

4.6 Connector Operations and Telemetry

#StepExpected ResultPass/FailNotes
4.6.1Check tracker health rollup APIReturns aggregated health across all connectors
4.6.2Verify ingress ops logsInbound webhook events logged with timestamps, connector ID
4.6.3Verify egress ops logsOutbound status sync calls logged with response codes
4.6.4Trigger a connector failureDead-letter telemetry records the failed event
4.6.5Verify dead-letter replayFailed event can be retried from ops view
4.6.6Verify rate limitingWebhook ingestion respects 300/min limit

Section 5: Spontaneous Stand-ups

5.1 Stand-up Trigger (LLM-Initiated)

#StepExpected ResultPass/FailNotes
5.1.1During a mission step, LLM reaches a decision point above significance thresholdStand-up triggered automatically
5.1.2Verify RACI inferenceCorrect participants selected (responsible, accountable, consulted, informed)
5.1.3Verify significance thresholdLow-significance decisions handled by LLM alone, not escalated
5.1.4Verify Slack notification sentStand-up thread created in designated Slack channel
5.1.5Verify SaaS decision inbox updatedPending decision appears in inbox

5.2 Stand-up Trigger (Human-Initiated)

#StepExpected ResultPass/FailNotes
5.2.1Human creates a stand-up from SaaS decision inboxStand-up session created with question context
5.2.2Slack notification sent to RACI participantsThread created with question, authority scope, response actions
5.2.3Verify participant list includes both humans and LLMsBoth types listed per RACI chart

5.3 Stand-up Discussion and Resolution

#StepExpected ResultPass/FailNotes
5.3.1Human responds in Slack threadResponse captured as collaboration event
5.3.2Multiple participants discussAll messages captured in timeline
5.3.3Mission owner makes final decisionDecision captured with: actor, timestamp, rationale, alternatives considered
5.3.4Verify decision feeds back to workflowOriginal LLM/mission step receives the decision and continues
5.3.5Verify audit trailFull stand-up history: trigger, participants, messages, decision, outcome

5.4 Stand-up Edge Cases

#StepExpected ResultPass/FailNotes
5.4.1Stand-up with no responses within timeoutTimeout escalation triggers (per runtime timeout policy)
5.4.2Mission owner unavailableEscalation path per RACI (accountable takes over)
5.4.3Duplicate stand-up for same decisionDeduplication prevents duplicate threads
5.4.4Stand-up while Slack disconnectedGraceful degradation, decision falls back to SaaS inbox only

Section 6: Payments and Billing (Polar)

6.1 Subscription Lifecycle

#StepExpected ResultPass/FailNotes
6.1.1Visit pricing pagePlans displayed with pricing, features, and CTA
6.1.2Start checkout for a paid planRedirected to Polar checkout
6.1.3Complete payment in Polar sandboxWebhook received, subscription state: active
6.1.4Verify team entitlements updatePaid features unlocked for team
6.1.5Add a team member (per-seat billing)Quantity incremented, Polar notified
6.1.6Remove a team memberQuantity decremented
6.1.7Simulate payment failureSubscription state: delinquent, grace period starts
6.1.8Resolve payment failureSubscription returns to active
6.1.9Cancel subscriptionState: canceled, access reverts at period end
6.1.10Verify audit eventsBilling lifecycle events logged with timestamps

6.2 Polar Webhook Handling

#StepExpected ResultPass/FailNotes
6.2.1Simulate checkout.completed webhookSubscription created, team mapped
6.2.2Simulate subscription.updated webhookSubscription state updated
6.2.3Simulate subscription.canceled webhookAccess scheduled for revocation
6.2.4Verify webhook signature validationInvalid signatures rejected
6.2.5Verify idempotent processingReplayed webhooks produce no side effects

Section 7: Transactional Emails

7.1 Account Emails (Existing)

#StepExpected ResultPass/FailNotes
7.1.1Sign upConfirmation email received
7.1.2Reset passwordReset email received with valid link
7.1.3Change emailVerification email sent to new address
7.1.4Team invitationInvitation email with accept link received

7.2 Workflow Notification Emails

#StepExpected ResultPass/FailNotes
7.2.1Decision request createdEmail sent to accountable/responsible participants
7.2.2Stand-up participation requestedEmail sent to RACI participants (if Slack unavailable)
7.2.3WP moved to for_reviewReviewer notification email sent
7.2.4WP moved to doneMission owner notification email sent
7.2.5Feature completed (all WPs done)Completion summary email to team
7.2.6Connector health degradedAdmin notification email sent

7.3 Email Preferences

#StepExpected ResultPass/FailNotes
7.3.1Navigate to email preferences pageAll notification categories listed with toggles
7.3.2Disable decision request emailsNo email sent on next decision request
7.3.3Re-enable decision request emailsEmail resumes on next decision request
7.3.4Verify team-level defaultsNew team members inherit team notification defaults
7.3.5User override of team defaultsPersonal toggle overrides team setting

Section 8: Status Parity (Local CLI vs SaaS)

8.1 Status Comparison

The bundled local web dashboard has been removed. Compare spec-kitty agent tasks status output with the SaaS dashboard for the same Mission:

#StepExpected ResultPass/FailNotes
8.1.1Compare lane layoutSame lanes, same lane names
8.1.2Compare WP contentSame information: WP ID, title, status, assignee
8.1.3Compare lane vocabularyIdentical: planned, claimed, in_progress, for_review, done, blocked, canceled
8.1.4Compare Mission list/overviewSame Missions visible with same status counts
8.1.5Take a SaaS screenshot and save the CLI --json outputArchive for brand review sign-off

8.2 Brand Token Verification

#StepExpected ResultPass/FailNotes
8.2.1Verify --sk-baby-blue (#A7C7E7) used in SaaSPresent in CSS custom properties
8.2.2Verify --sk-grassy-green (#7BB661) mapped to --primaryDaisyUI primary uses grassy-green family
8.2.3Verify neutral surfacesCreamy-white/light-gray family for backgrounds
8.2.4Verify --sk-dark-text (#2c3e50)Body text uses spec-kitty dark text
8.2.5Verify DaisyUI error/success semantics preservedError/success badges still use DaisyUI defaults for accessibility
8.2.6Check contrast ratiosBadge + text combinations meet WCAG AA

Section 9: API and Integration Contracts

9.1 REST API

#StepExpected ResultPass/FailNotes
9.1.1Fetch OpenAPI schema: GET /api/schema/Valid OpenAPI 3.0 document returned
9.1.2Open Swagger UI: /api/schema/swagger-ui/Interactive API documentation loads
9.1.3Open ReDoc: /api/schema/redoc/Reference documentation loads
9.1.4GET /api/v1/sync/projects/ with JWTReturns team-scoped project list
9.1.5GET /api/v1/sync/features/ with JWTReturns features for project
9.1.6GET /api/v1/sync/work-packages/ with JWTReturns WPs with status
9.1.7POST /api/v1/events/batch/ with event arrayEvents ingested, 2xx response
9.1.8Verify team isolation on APICannot access other team’s resources with valid JWT
9.1.9Verify rate limitingExceeding rate limit returns 429
9.1.10Use API key (UserAPIKey) instead of JWTSame endpoints accessible
9.1.11Use TeamAPIKeyMachine-client access works for team resources

9.2 WebSocket

#StepExpected ResultPass/FailNotes
9.2.1Connect WebSocket with valid tokenConnection established
9.2.2Verify heartbeatPing/pong within expected interval
9.2.3Send event via WebSocketEvent processed, acknowledged
9.2.4Receive real-time updateStatus change from CLI appears via WebSocket push
9.2.5Connect with expired tokenRejected with auth error

Section 10: Monitoring and Observability

10.1 Health Checks

#StepExpected ResultPass/FailNotes
10.1.1GET /health/Returns 200 with database and Redis status
10.1.2Kill Redis, check healthReturns degraded status for Redis
10.1.3Restart Redis, check healthReturns healthy

10.2 Prometheus Metrics

#StepExpected ResultPass/FailNotes
10.2.1GET /metricsPrometheus-format metrics returned
10.2.2Verify request timing metricsdjango_http_requests_latency_seconds present
10.2.3Verify custom business metricsEvent ingestion counts, sync latency present

10.3 Sentry Integration

#StepExpected ResultPass/FailNotes
10.3.1Trigger an unhandled exceptionError appears in Sentry with full stack trace
10.3.2Verify user contextSentry event includes user ID and team
10.3.3Verify tag propagationCustom tags (project_slug, feature_slug) present

Section 11: Security

11.1 Authentication Security

#StepExpected ResultPass/FailNotes
11.1.1Attempt SQL injection on login formRejected, no error leak
11.1.2Attempt XSS in user-submitted contentContent sanitized, no script execution
11.1.3Attempt CSRF on state-changing endpointRejected without valid CSRF token
11.1.4Verify credential file permissions~/.spec-kitty/credentials has 0600 permissions
11.1.5Verify connector secrets encrypted at restFernet-encrypted in database, not plaintext

11.2 Authorization

#StepExpected ResultPass/FailNotes
11.2.1Member attempts admin actionRejected with 403
11.2.2User attempts to access another team’s dataRejected, no data leak
11.2.3API key with revoked team accessRejected immediately
11.2.4Expired JWT without refresh401 returned, no access

Section 12: Edge Cases and Error Handling

12.1 Concurrent Agent Operations

#StepExpected ResultPass/FailNotes
12.1.1Two agents implement same WP simultaneouslyConcurrent driver warning emitted
12.1.2Two agents move same WP to different lanesLast-write-wins with full audit trail, no data corruption
12.1.3Agent works in stale worktreeClear error or merge conflict on push, not silent data loss

12.2 Network Failures

#StepExpected ResultPass/FailNotes
12.2.1CLI loses network mid-syncGraceful degradation to offline queue
12.2.2SaaS database unreachableHealth check fails, 503 returned, no crash
12.2.3Connector webhook endpoint unreachableDead-letter queue captures event for retry
12.2.4Nango token refresh failsConnector health degrades, ops log records it

12.3 Data Integrity

#StepExpected ResultPass/FailNotes
12.3.1Corrupt a line in status.events.jsonlspec-kitty doctor mission-state --audit detects corruption, reports it
12.3.2Duplicate event IDs in event logDeduplication prevents double-processing
12.3.3Merge two branches with divergent event logsEvents merged deterministically by ULID ordering
12.3.4Delete a WP file while in in_progressStatus system detects orphan, doctor mission-state --audit reports it

Section 13: Performance Smoke Tests

#StepExpected ResultPass/FailNotes
13.1SaaS dashboard page load (cold)Under 3 seconds
13.2SaaS dashboard page load (warm)Under 1 second
13.3Kanban board with 50+ WPsRenders without lag
13.4Batch sync of 100 queued eventsCompletes within 10 seconds
13.6spec-kitty status commandOutput in under 1 second
13.7spec-kitty next commandDecision returned in under 2 seconds
13.8WebSocket reconnection after dropReconnects within 5 seconds (first attempt)

Section 14: GA Readiness Gate Checklist

Cross-reference with Feature 022 WP09 GA readiness report:

GateCriterionStatusNotes
BillingPolar checkout → subscription → entitlement flow works
BillingSubscription lifecycle (trial → active → delinquent → canceled)
OAuth (Jira)Connect → token → sync → rotate → disconnect
OAuth (Linear)Connect → token → sync → rotate → disconnect
OAuth (GitHub)Connect → token → webhook → sync → disconnect
OAuth (GitLab)Connect → token → webhook → sync → disconnect
OAuth (Slack)Connect → bot → thread → notification → disconnect
NotificationsDecision request email triggers
NotificationsStand-up participation email triggers
NotificationsStatus transition notifications trigger
NotificationsUser preference toggles work
DashboardBrand tokens applied
DashboardStatus parity with local CLI verified (Section 8)
SecurityConnector secrets encrypted at rest
SecurityWebhook signature validation on all endpoints
ObservabilityConnector health rollup API operational
ObservabilityIngress/egress ops logs visible
ObservabilityDead-letter telemetry operational
StabilityNo critical Sentry errors in 48-hour staging soak

Execution Notes

Test Order Recommendation

  1. Section 1 (CLI core) — baseline verification, no SaaS needed
  2. Section 2 (Auth/sync) — establishes CLI ↔ SaaS connection
  3. Section 3 (SaaS dashboard) — verify the web experience
  4. Section 8 (Status parity) — compare local CLI vs SaaS
  5. Section 4.1–4.4 (Jira → Linear → GitHub → GitLab) — follows ADR connector order
  6. Section 4.5 (Slack) — prerequisite for stand-ups
  7. Section 5 (Spontaneous stand-ups) — the flagship differentiator
  8. Section 6 (Payments) — Polar integration
  9. Section 7 (Emails) — notification flows
  10. Section 9–13 (API, monitoring, security, edge cases, performance) — hardening

Reporting

For each section, record:

  • Date tested
  • Tester name
  • Environment (staging URL, CLI version, browser)
  • Pass/Fail for each line item
  • Screenshots for visual tests (Sections 3, 8)
  • Blockers requiring engineering fix before retest

Retest Policy

  • Any Fail in Sections 1–6 blocks Beta
  • Any Fail in Section 14 (GA gate) blocks Paid GA
  • Section 12 (edge cases) failures are logged as issues, not Beta blockers unless data loss
  • Section 13 (performance) failures trigger investigation, not automatic block