LLM Routing Cost Reduction: Cut AI Bills 67% with Smart Model Tiers

You're Probably Overpaying for LLM Calls

Most teams default to routing all traffic to their flagship model after the initial integration. "Improve my typo" goes to Claude Opus. "What's the weather intent?" hits GPT-5.5. That's not using a sharp knife — that's running a nuclear plant to boil a kettle.

Flagship model pricing typically runs 10–20× higher than lightweight alternatives. At 10,000+ daily requests, the cost delta stops being a line item and starts being a meeting agenda.

The core principle behind LLM routing cost reduction is simple: use the cheapest model that can solve the problem, not the best one available. The hard part isn't picking models — it's accurately judging the complexity of each incoming request.

Four Dimensions of Request Complexity

Before building a classifier, you need to define what "complexity" actually means in measurable terms. Four dimensions reliably predict how much model horsepower a request needs:

1. Intent type — casual chat, information lookup, reasoning/analysis, code generation, multi-step planning. Difficulty increases in that order.
2. Context length — conversation histories over 8K tokens demand stronger long-range comprehension.
3. Output structure — plain text versus executable JSON, SQL, or code. Structured outputs have low error tolerance; mistakes are expensive.
4. Domain depth — general Q&A versus legal compliance, medical diagnosis, or complex financial modeling. Hallucination costs are asymmetric in high-stakes domains.

Score each dimension 0–3. Total scores 0–3 go to a lightweight model, 4–7 to a balanced model, 8–12 to flagship. That's the skeleton of tiered routing.

The Classifier: Two Layers

A complete classifier uses two layers: a rule layer that catches obviously simple or complex requests, and a model layer that handles the ambiguous middle ground.

Rule layer (zero latency, zero cost):

SIMPLE_PATTERNS = [
    r"^(hi|hello|hey|thanks|ok|sure)",
    r"translate.{0,20}(to|into)",          # short text translation
    r"(fix typo|proofread.{0,10}sentence)",
    r"^(today|tomorrow|now).{0,15}(weather|time|date)",
]

COMPLEX_SIGNALS = [
    r"(analyze|evaluate|compare).{0,30}(plan|strategy|risk)",
    r"(write|generate|implement).{0,20}(code|script|algorithm)",
    r"(legal|compliance|regulatory|litigation)",
    r"(reasoning|step.by.step|chain.of.thought)",
]

def rule_classify(text: str) -> str | None:
    for p in SIMPLE_PATTERNS:
        if re.search(p, text, re.I): return "simple"
    for p in COMPLEX_SIGNALS:
        if re.search(p, text, re.I): return "complex"
    return None  # pass to model layer

Model layer (uses the cheapest model to label requests — cost is negligible):

CLASSIFIER_PROMPT = """
You are a request complexity classifier. Return JSON only, no explanation.
Scoring:
- intent_score: 0(chat) 1(lookup) 2(analysis) 3(planning/code)
- context_score: 0(<2K) 1(2-8K) 2(8-32K) 3(>32K tokens)
- structure_score: 0(plain text) 1(list) 2(structured JSON) 3(executable code)
- domain_score: 0(general) 1(professional) 2(high-risk) 3(critical-risk)
Output: {"total": <sum>, "tier": "simple|balanced|flagship"}
"""

async def model_classify(text: str, ctx_tokens: int) -> dict:
    resp = await client.chat.completions.create(
        model="claude-haiku-4-5",
        messages=[
            {"role": "system", "content": CLASSIFIER_PROMPT},
            {"role": "user", "content": f"Request: {text[:500]}\nContext tokens: {ctx_tokens}"}
        ],
        max_tokens=60,
        temperature=0
    )
    return json.loads(resp.choices[0].message.content)

Combined routing logic:

MODEL_MAP = {
    "simple":   "claude-haiku-4-5",       # ~$0.25/M input tokens
    "balanced": "claude-sonnet-4-6",      # ~$3/M input tokens
    "flagship": "claude-opus-4-8",        # ~$15/M input tokens
}

async def route(text: str, ctx_tokens: int) -> str:
    tier = rule_classify(text) or (await model_classify(text, ctx_tokens))["tier"]
    return MODEL_MAP[tier]

Real-World Cost Numbers

Here's a concrete example: a mixed customer service and content generation workload, 50,000 requests per day, previously running everything through Claude Sonnet 4.6.

Tier	Share	Request types	Model	Price (input/M)
simple	52%	Greetings, typo fixes, short translations	Haiku 4.5	$0.25
balanced	35%	Summaries, FAQ answers, light code	Sonnet 4.6	$3.00
flagship	13%	Complex reasoning, compliance review, architecture	Opus 4.8	$15.00

At 100M input tokens per day, flat Sonnet 4.6 costs $300/day. The naive weighted average after routing gives $3.13/M — but that ignores that simple requests are also shorter. They average roughly a quarter of the token volume of complex requests. After correcting for that, actual daily cost drops to around $98 — a 67% reduction.

Latency improves in parallel. Haiku 4.5 typically delivers a first token in 200–400ms versus 800–1500ms for Opus 4.8. Routing 52% of requests to Haiku means those users see responses 3–4× faster, with no quality tradeoff.

Keeping the Classifier Honest

Routing errors are asymmetric. Sending a complex request to a lightweight model (under-classification) causes quality failures. Sending a simple request to a flagship model (over-classification) just costs money. Set your threshold conservatively: escalate to flagship at a total score ≥ 6, not ≥ 8.

Track two metrics continuously in production:

• Downgrade failure rate — requests classified as simple that triggered retries or negative feedback. If this exceeds 3%, recalibrate the rule layer.
• Classification latency — rule layer should be under 1ms; model layer should stay under 300ms.

One optimization that's easy to overlook: classification and the main request don't have to run sequentially. Fire both in parallel — pre-warm the balanced model connection while classification runs. When the classification result arrives, switch to the right target. This removes most of the model-layer classification time from the critical path.

Extending to Multi-Provider Routing

Tiered routing doesn't have to be single-vendor. Within each tier, you can add latency-based fallbacks: if Opus 4.8 is rate-limited, automatically switch to GPT-5.5; if Haiku 4.5 is down, fall back to GPT-5.4-mini or a lightweight Qwen 3 variant. This only works cleanly if your API layer uses OpenAI-compatible format with a unified model field — your application code stays completely unchanged.

FALLBACK_MAP = {
    "claude-opus-4-8":   ["gpt-5.5", "gemini-3-ultra"],
    "claude-sonnet-4-6": ["gpt-5.4", "gemini-3-pro"],
    "claude-haiku-4-5":  ["gpt-5.4-mini", "qwen3-8b"],
}

This architecture makes a real engineering tradeoff across cost, quality, and latency. The goal isn't "always use the best" — it's "use exactly what the job requires."

---

Once this routing layer is in place, you need a stable multi-model access layer to run it on. I've been using XycAi (https://www.xyc.ai) — a single OpenAI-compatible API covering 200+ global models, including all the Claude, GPT-5 series, Gemini 3, and Qwen 3 variants mentioned above. Flagship models start at 14% of list price, which puts another dent in the flagship tier costs even after routing. Claude Code and Codex are also supported with one-command setup.