How to Build a Real AI Customer Service Chatbot API (Not Just a Chat Box)

Most "AI customer service" is just a chat box

A lot of teams wire an LLM to a chat window, ship it, and call it an intelligent support system. Then a user asks "where's my order?" and the model confidently fabricates shipping details. A frustrated user sends an angry message and gets a polished non-answer in return.

The model isn't the problem. The architecture is.

A support system that actually works needs to solve three things: know what the user is asking (intent classification), know where to find the answer (knowledge retrieval), and know when to hand off to a human (escalation logic). Miss any one of them and you get systematic failure on an entire category of questions.

---

Intent classification: stop letting the model guess

The most common mistake is routing every message directly to a flagship model — Claude Opus or GPT-4o — and hoping it figures out what to do. That's expensive, slow, and gives you no measurable accuracy on intent.

A better approach is a lightweight classification layer at the front. Use a small model like Claude Haiku or GPT-4o-mini for intent classification. The cost is roughly 1/10 of a flagship call, and you can keep P50 latency under 300ms. The classification result drives all downstream routing.

A typical intent classification prompt:

INTENT_SYSTEM = """
You are a customer service intent classifier. Assign the user message to exactly one category and return only JSON:
- order_query: order status, shipping inquiries
- refund_request: refund or return requests
- product_consult: product features, how-to questions
- complaint: complaints, negative sentiment
- human_request: user explicitly asks for a human agent
- other: cannot be classified

Output format: {"intent": "<category>", "confidence": 0.0-1.0}
"""

async def classify_intent(user_message: str) -> dict:
    response = await client.chat.completions.create(
        model="claude-haiku-4-5",  # or gpt-4o-mini
        messages=[
            {"role": "system", "content": INTENT_SYSTEM},
            {"role": "user", "content": user_message}
        ],
        max_tokens=60,
        temperature=0
    )
    return json.loads(response.choices[0].message.content)

When confidence comes back below 0.6, fall back — ask a clarifying question or escalate to a human. Never hard-guess a low-confidence intent.

---

RAG pipeline: the parameters that actually matter

Once intent is clear, most questions need answers pulled from a knowledge base, not generated from the model's memory. RAG is the standard pattern here, but implementation quality varies enormously.

Chunking and retrieval

Chunk your knowledge base at 512 tokens with a 64-token overlap. For embeddings, text-embedding-3-large or an equivalent works well. At query time, retrieve the top 5 candidates, then rerank with BM25 hybrid search down to top 3. Pure vector search struggles with exact-match queries — order numbers, SKUs, product codes. Hybrid search pushes recall from around 72% to roughly 88% on those cases.

Assembling the prompt

Inject retrieved documents and conversation history together. Wrap knowledge base content in XML tags so the model doesn't blend sources:

def build_rag_prompt(retrieved_docs: list, history: list, user_query: str) -> list:
    knowledge_block = "\n".join([
        f"<doc id='{i}'>{doc['content']}</doc>"
        for i, doc in enumerate(retrieved_docs)
    ])

    system = f"""You are a customer service assistant. Answer only using the knowledge base below.
If the answer isn't there, say "Let me connect you with a specialist to confirm that." Do not make anything up.

<knowledge_base>
{knowledge_block}
</knowledge_base>"""

    messages = [{"role": "system", "content": system}]
    messages.extend(history[-6:])  # last 3 turns
    messages.append({"role": "user", "content": user_query})
    return messages

Keeping 6 messages (3 turns) of history hits the practical sweet spot. Beyond that, context costs grow linearly but response quality barely moves.

---

Multi-turn state management

Support conversations don't follow a clean single-turn structure. Users switch topics mid-conversation. They explain their problem across three messages. Stateless API calls handle this badly — the system "forgets" constantly.

The fix is a session object stored in Redis with a 30-minute TTL:

@dataclass
class SessionState:
    session_id: str
    user_id: str
    history: list[dict]          # conversation history
    current_intent: str          # active intent
    context: dict                # cross-turn data like order numbers
    escalation_signals: int      # counter for escalation scoring
    created_at: float
    last_active: float

When intent is order_query, the system extracts and stores the order number in context. Subsequent turns don't need the user to repeat it. escalation_signals feeds directly into the handoff logic.

---

Human escalation: rules plus a sentiment layer

Escalation is the part most systems get wrong. Too aggressive and the AI adds no value; too conservative and frustrated users hit a wall.

The pattern that works: a rule layer combined with async sentiment detection.

Rule layer — escalate immediately if any of these are true: - Intent is human_request - Intent is complaint and escalation_signals >= 2 - Three consecutive turns with RAG retrieval confidence below threshold - Refund or transaction amount exceeds a business-defined limit

Sentiment detection runs asynchronously after each reply — it doesn't block the main response:

async def detect_sentiment(message: str) -> float:
    """Returns a 0–1 negative sentiment score"""
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"Rate the negative sentiment intensity in this customer service message from 0 to 1. Return only the number:\n{message}"
        }],
        max_tokens=5,
        temperature=0
    )
    return float(resp.choices[0].message.content.strip())

A score above 0.75 increments escalation_signals. Each check costs 50–100 tokens — negligible at scale.

When escalation triggers, the system passes the full SessionState to the human agent: conversation history, extracted context, identified intent. The agent doesn't have to ask the user to start over. Most systems skip this step. It's the single biggest factor in whether escalation feels smooth or broken.

---

Three-tier model routing

The underlying cost logic for this architecture:

Tier	Model	Avg latency	Approx cost per call
Intent classification	Haiku / GPT-4o-mini	~300ms	~$0.0001
RAG Q&A	Sonnet / GPT-4o	~800ms	~$0.002
Complex complaints	Opus / GPT-4	~2000ms	~$0.015

Flagship models only handle cases that genuinely need deep reasoning. The majority of conversations route through the middle tier. Total cost lands around 20% of a flat flagship-only setup.

---

For a system like this, model-switching flexibility matters as much as the architecture itself. I use XycAi to manage multi-model access across the stack — one OpenAI-compatible API covering 200+ models, with Claude and GPT pricing starting at 14% of list price. For a project where you're constantly testing different model combinations across tiers, the unified interface and the cost difference add up fast.