<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" xml:base="https://docs.spec-kitty.ai/changelog/2x/model-discipline-routing/">
  <id>https://docs.spec-kitty.ai/changelog/2x/model-discipline-routing/</id>
  <title>2.x Model Discipline and Cost-Aware Routing (Draft)</title>
  <updated>2026-06-03T00:00:00.000Z</updated>
  <author><name>Spec Kitty</name></author>
  <link rel="alternate" type="text/html" href="https://docs.spec-kitty.ai/changelog/2x/model-discipline-routing/"/>
  <summary>Historical Spec Kitty 2.x archive page for 2.x Model Discipline and Cost-Aware Routing (Draft); use Spec Kitty 3.2 docs for current Charter-era workflows.</summary>
  <content type="html">&lt;blockquote&gt;
&lt;p&gt;Archive notice: This page documents historical Spec Kitty behavior and is not the current 3.2 workflow. Start with &lt;a href=&quot;/context/&quot;&gt;Spec Kitty 3.2&lt;/a&gt; for current docs.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;span id=&quot;2x-model-discipline-and-cost-aware-routing-draft&quot;&gt;&lt;/span&gt;
&lt;p&gt;&lt;strong&gt;Status&lt;/strong&gt;: Draft&lt;br&gt;
&lt;strong&gt;Date&lt;/strong&gt;: 2026-02-28&lt;br&gt;
&lt;strong&gt;Scope&lt;/strong&gt;: User journey + implementation research for model-aware task assignment&lt;/p&gt;
&lt;h2 id=&quot;problem-statement&quot;&gt;Problem Statement&lt;/h2&gt;
&lt;p&gt;In current 2.x flow, task execution is agent-selected by the operator (&lt;code dir=&quot;auto&quot;&gt;--agent &amp;#x3C;name&gt;&lt;/code&gt;), with optional static preferences in
&lt;code dir=&quot;auto&quot;&gt;.kittify/config.yaml&lt;/code&gt;. This is simple, but it does not optimize for:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Task fit (which model is strongest for this task type)&lt;/li&gt;
&lt;li&gt;Quality/cost trade-offs&lt;/li&gt;
&lt;li&gt;Consistent governance when assignment decisions vary by operator&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;current-baseline-what-exists-today&quot;&gt;Current Baseline (What Exists Today)&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;spec-kitty next&lt;/code&gt; requires &lt;code dir=&quot;auto&quot;&gt;--agent&lt;/code&gt; and does not recommend a model/tool automatically (&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/cli/commands/next_cmd.py&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Tool selection uses the &lt;code dir=&quot;auto&quot;&gt;agents.available&lt;/code&gt; list order with first-available fallback (
&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/core/tool_config.py&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Agent profiles already support weighted matching by task context, but no model-cost dimension (&lt;code dir=&quot;auto&quot;&gt;src/doctrine/agent_profiles/repository.py&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Doctrine artifacts already provide governance hooks (directives, toolguides) for 2.x (&lt;code dir=&quot;auto&quot;&gt;docs/changelog/2x/doctrine-and-charter.md&lt;/code&gt;).&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;proposed-user-journey&quot;&gt;Proposed User Journey&lt;/h2&gt;
&lt;h3 id=&quot;scenario&quot;&gt;Scenario&lt;/h3&gt;
&lt;p&gt;A team enables a &lt;strong&gt;Model Discipline&lt;/strong&gt; doctrine rule so Spec Kitty can recommend the best available model for each task, balancing quality, cost tier, and known weaknesses, while still allowing explicit operator override.&lt;/p&gt;
&lt;h3 id=&quot;actors&quot;&gt;Actors&lt;/h3&gt;



































&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;#&lt;/th&gt;&lt;th&gt;Actor&lt;/th&gt;&lt;th&gt;Type&lt;/th&gt;&lt;th&gt;Role in Journey&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;Project Operator&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;human&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Chooses policy level and can override recommendations&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;2&lt;/td&gt;&lt;td&gt;Spec Kitty CLI&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;system&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Computes task type, recommends tool/model pair, enforces policy&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;3&lt;/td&gt;&lt;td&gt;Model Catalog Updater&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;system&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Refreshes capability/cost metadata from configured sources&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;4&lt;/td&gt;&lt;td&gt;Agent Runtime (Claude/Codex/etc.)&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;llm&lt;/code&gt;&lt;/td&gt;&lt;td&gt;Executes the assigned work package&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;h3 id=&quot;preconditions&quot;&gt;Preconditions&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Feature and work packages exist (&lt;code dir=&quot;auto&quot;&gt;kitty-specs/&amp;#x3C;mission&gt;/tasks/WP*.md&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;.kittify/config.yaml&lt;/code&gt; has available tools configured.&lt;/li&gt;
&lt;li&gt;Doctrine bundle includes model-discipline directive/toolguide and task-type mapping file.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;journey-map&quot;&gt;Journey Map&lt;/h3&gt;















































&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Phase&lt;/th&gt;&lt;th&gt;Actor(s)&lt;/th&gt;&lt;th&gt;System&lt;/th&gt;&lt;th&gt;Key Events&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;1. Configure Policy&lt;/td&gt;&lt;td&gt;Operator&lt;/td&gt;&lt;td&gt;Enables model-discipline mode in config/charter profile&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;ModelDisciplineConfigured&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;2. Refresh Catalog&lt;/td&gt;&lt;td&gt;Model Catalog Updater&lt;/td&gt;&lt;td&gt;Pulls latest model ranking metadata and pricing snapshots&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;ModelCatalogRefreshed&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;3. Classify Task&lt;/td&gt;&lt;td&gt;Spec Kitty CLI&lt;/td&gt;&lt;td&gt;Determines task type from mission step + WP metadata&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;TaskTypeClassified&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;4. Recommend Assignment&lt;/td&gt;&lt;td&gt;Spec Kitty CLI&lt;/td&gt;&lt;td&gt;Scores candidates by quality/cost/risk and suggests tool+model&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;ModelAssignmentRecommended&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;5. Execute/Override&lt;/td&gt;&lt;td&gt;Operator + Agent Runtime&lt;/td&gt;&lt;td&gt;Accepts recommendation or overrides with reason&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;ModelAssignmentAccepted&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;ModelAssignmentOverridden&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;6. Capture Metrics&lt;/td&gt;&lt;td&gt;Spec Kitty CLI&lt;/td&gt;&lt;td&gt;Stores usage/cost/performance outcomes for future tuning&lt;/td&gt;&lt;td&gt;&lt;code dir=&quot;auto&quot;&gt;ModelExecutionMetricsCaptured&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;h3 id=&quot;coordination-rules&quot;&gt;Coordination Rules&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Default posture&lt;/strong&gt;: Advisory (Phase 1), then Gated (Phase 2+)&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;If policy mode is &lt;code dir=&quot;auto&quot;&gt;advisory&lt;/code&gt;, non-compliant selections warn but do not block.&lt;/li&gt;
&lt;li&gt;If policy mode is &lt;code dir=&quot;auto&quot;&gt;gated&lt;/code&gt;, non-compliant selections require explicit override reason.&lt;/li&gt;
&lt;li&gt;If policy mode is &lt;code dir=&quot;auto&quot;&gt;required&lt;/code&gt;, assignment blocks until a compliant model or override waiver is recorded.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;proposed-2x-artifact-design&quot;&gt;Proposed 2.x Artifact Design&lt;/h2&gt;
&lt;h3 id=&quot;1-new-directive&quot;&gt;1. New Directive&lt;/h3&gt;
&lt;p&gt;Add a doctrine directive such as:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/doctrine/directives/020-model-discipline-routing.directive.yaml&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Purpose:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Require model-to-task fit checks before assignment.&lt;/li&gt;
&lt;li&gt;Require cost-tier awareness and explicit override capture.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;2-new-toolguide&quot;&gt;2. New Toolguide&lt;/h3&gt;
&lt;p&gt;Add a model discipline toolguide:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/doctrine/toolguides/model-discipline.toolguide.yaml&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/doctrine/toolguides/MODEL_DISCIPLINE.md&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Purpose:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Define task type taxonomy (&lt;code dir=&quot;auto&quot;&gt;implementation&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;review&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;research&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;refactor&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;doc-authoring&lt;/code&gt;, etc.).&lt;/li&gt;
&lt;li&gt;Explain scoring dimensions (quality, weakness risk, cost tier, latency tier).&lt;/li&gt;
&lt;li&gt;Define override rules and audit expectations.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;3-model-to-task-mapping-data&quot;&gt;3. Model-to-Task Mapping Data&lt;/h3&gt;
&lt;p&gt;Add a machine-readable mapping file:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/doctrine/toolguides/model-to-task_type.yml&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Suggested schema sections:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;task_types&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;models&lt;/code&gt; (strengths, weaknesses, supported tools, cost tier, optional price per 1M tokens)&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;routing_policy&lt;/code&gt; (weights + thresholds)&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;sources&lt;/code&gt; (where each metric came from + timestamp)&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Recommended follow-up:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Add &lt;code dir=&quot;auto&quot;&gt;src/doctrine/schemas/model-to-task_type.schema.yaml&lt;/code&gt; for validation.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;proposed-yaml-schema-v10&quot;&gt;Proposed YAML Schema (v1.0)&lt;/h2&gt;
&lt;p&gt;Use this schema for validating &lt;a href=&quot;/changelog/2x/model-to-task_type/&quot;&gt;&lt;code dir=&quot;auto&quot;&gt;model-to-task_type.yml&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;
&lt;h2 id=&quot;implementation-approach-phased&quot;&gt;Implementation Approach (Phased)&lt;/h2&gt;
&lt;h3 id=&quot;phase-0-advisory-only-research-slice&quot;&gt;Phase 0: Advisory-only research slice&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Load mapping file and compute recommendations without changing assignment behavior.&lt;/li&gt;
&lt;li&gt;Expose recommendation in &lt;code dir=&quot;auto&quot;&gt;spec-kitty next --json&lt;/code&gt; payload as additional fields (&lt;code dir=&quot;auto&quot;&gt;recommended_agent&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;recommended_model&lt;/code&gt;, &lt;code dir=&quot;auto&quot;&gt;rationale&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Keep &lt;code dir=&quot;auto&quot;&gt;--agent&lt;/code&gt; mandatory for backwards compatibility.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;phase-1-optional-auto-selection&quot;&gt;Phase 1: Optional auto-selection&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Add &lt;code dir=&quot;auto&quot;&gt;--agent auto&lt;/code&gt; support in &lt;code dir=&quot;auto&quot;&gt;next&lt;/code&gt; and workflow commands.&lt;/li&gt;
&lt;li&gt;Resolve &lt;code dir=&quot;auto&quot;&gt;tool+model&lt;/code&gt; from mapping and configured availability.&lt;/li&gt;
&lt;li&gt;Persist selected model in WP history metadata (or event payload) for traceability.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;phase-2-directive-enforcement&quot;&gt;Phase 2: Directive enforcement&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Wire directive checks into execution path (pre-transition validation before moving WP to &lt;code dir=&quot;auto&quot;&gt;doing&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Enforce advisory/gated/required behavior from config.&lt;/li&gt;
&lt;li&gt;Require override reason when non-compliant choices are used.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;phase-3-feedback-loop&quot;&gt;Phase 3: Feedback loop&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Record observed cost, latency, and acceptance outcomes.&lt;/li&gt;
&lt;li&gt;Use telemetry to tune weights and reduce static assumptions.&lt;/li&gt;
&lt;li&gt;Add periodic catalog sync command (e.g., &lt;code dir=&quot;auto&quot;&gt;spec-kitty model sync&lt;/code&gt;).&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;integration-points-in-current-code&quot;&gt;Integration Points in Current Code&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/core/tool_config.py&lt;/code&gt;&lt;br&gt;
Extend selection config with model-discipline routing dimensions.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/cli/commands/next_cmd.py&lt;/code&gt;&lt;br&gt;
Add optional auto-routing input/output surface.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/next/runtime_bridge.py&lt;/code&gt;&lt;br&gt;
Compute routing recommendation before decision output.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/specify_cli/cli/commands/agent/workflow.py&lt;/code&gt;&lt;br&gt;
Consume resolved agent/model for implement/review handoff.&lt;/li&gt;
&lt;li&gt;&lt;code dir=&quot;auto&quot;&gt;src/doctrine/agent_profiles/repository.py&lt;/code&gt;&lt;br&gt;
Extend weighted scoring with cost/quality dimensions from model catalog.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;data-source-research-notes&quot;&gt;Data Source Research Notes&lt;/h2&gt;
&lt;h3 id=&quot;arena-data&quot;&gt;Arena data&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Arena’s public docs describe leaderboard evaluation as human preference voting, and publish open datasets monthly.&lt;/li&gt;
&lt;li&gt;Arena Terms include restrictions on bots/scraping/harvesting without authorization.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Implication:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Prefer documented/official data channels (published datasets/APIs) over raw HTML scraping of &lt;code dir=&quot;auto&quot;&gt;arena.ai/leaderboard&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Treat Arena rankings as one quality signal, not a direct task-specialization oracle.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;cost-data&quot;&gt;Cost data&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Arena leaderboard data does not provide comprehensive pricing metadata.&lt;/li&gt;
&lt;li&gt;Cost-per-1M token fields should be sourced from provider pricing pages/APIs and versioned with timestamps.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;risks-and-mitigations&quot;&gt;Risks and Mitigations&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Config schema drift&lt;/strong&gt;: current save path rewrites &lt;code dir=&quot;auto&quot;&gt;tools.selection&lt;/code&gt; with only two fields.&lt;br&gt;
Mitigation: introduce explicit schema version + preserve/round-trip unknown fields during migration.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;False confidence from leaderboard rank&lt;/strong&gt;: global rank may not match local task quality.&lt;br&gt;
Mitigation: include weakness flags and local telemetry feedback.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Operator trust&lt;/strong&gt;: opaque routing decisions reduce adoption.&lt;br&gt;
Mitigation: always show rationale and allow controlled override.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rapid model churn&lt;/strong&gt;: rankings and prices change frequently.&lt;br&gt;
Mitigation: freshness timestamps + configurable staleness thresholds.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;suggested-mvp-decision&quot;&gt;Suggested MVP Decision&lt;/h2&gt;
&lt;p&gt;Start with &lt;strong&gt;advisory mode&lt;/strong&gt; in 2.x:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Add directive + toolguide + mapping file.&lt;/li&gt;
&lt;li&gt;Return recommendation metadata in &lt;code dir=&quot;auto&quot;&gt;next --json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Do not block current workflows yet.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This gives immediate value (visibility + consistency) with low migration risk.&lt;/p&gt;
&lt;h2 id=&quot;external-references&quot;&gt;External References&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;Arena home/terms: &lt;a href=&quot;https://lmarena.ai/&quot;&gt;https://lmarena.ai/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Arena how-it-works and dataset notes: &lt;a href=&quot;https://lmarena.ai/howitworks/&quot;&gt;https://lmarena.ai/howitworks/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Arena leaderboard policy: &lt;a href=&quot;https://lmarena.ai/leaderboard-policy/&quot;&gt;https://lmarena.ai/leaderboard-policy/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Arena-rank package (public ranking/data access workflow): &lt;a href=&quot;https://pypi.org/project/arena-rank/&quot;&gt;https://pypi.org/project/arena-rank/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;OpenAI API pricing: &lt;a href=&quot;https://openai.com/api/pricing/&quot;&gt;https://openai.com/api/pricing/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Anthropic API pricing: &lt;a href=&quot;https://www.anthropic.com/pricing&quot;&gt;https://www.anthropic.com/pricing&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;</content>
</entry>
