Chain of Thought Prompting Guide: CoT, Zero-Shot & ToT

Why "Show Your Work" Makes Models More Accurate

Most people discover chain of thought prompting the same way: a model confidently gets a logic problem wrong, you add "think step by step" to the prompt, and suddenly it's right. That gap is worth understanding — not just as a party trick, but as a window into how these models actually process information.

Language models generate text one token at a time, and each token is predicted from everything that came before it. When you ask "if A > B and B > C, which is larger — A or C?" the model jumps straight from question to answer with no working space. But when it first writes out "A > B, B > C, therefore A > C" — that reasoning chain becomes part of the context for the final answer. It's essentially working memory the model created for itself.

The original CoT paper from Google (Wei et al., 2022) demonstrated this on GSM8K, a benchmark of grade-school math problems: standard few-shot prompting hit about 17% accuracy; adding chain of thought examples pushed it to 56%. Modern models have raised those absolute numbers considerably, but the relative lift from CoT remains consistent — especially on tasks that require multiple reasoning steps.

This guide covers the three main CoT variants, when to use each, and templates you can drop straight into your prompts.

---

The Three Main Variants

Few-Shot CoT

This is the original form. You include three to eight worked examples in your prompt — each showing the full reasoning process before the answer — and the model learns the "reason first, then answer" pattern from those examples.

Best for: math word problems, logical deduction, code debugging, and any domain where you can write high-quality examples in advance and the problem structure is consistent enough to make those examples reusable.

The tradeoff: you're paying for those examples in tokens on every single request. At scale, with models like GPT-4o or Claude Sonnet, that adds up fast.

Template:

Here are some worked examples:

Question: Maya has 12 apples. She gives away 1/3 of them, then buys 5 more. How many does she have?
Reasoning: 12 × 1/3 = 4. She gives away 4, leaving 8. Then 8 + 5 = 13.
Answer: 13

---
Now solve this:
Question: {{your question here}}
Reasoning:

---

Zero-Shot CoT

In 2022, Kojima et al. discovered something surprisingly simple: you don't need examples at all. Just append "Let's think step by step" to your question and most capable models will spontaneously produce a reasoning chain.

What this means in practice is that the responsibility shifts from your prompt to the model. You're trusting the model's own internalized reasoning patterns rather than steering them with examples. On models that have been trained on large amounts of CoT data — which describes most frontier models today — zero-shot CoT often gets you 80–90% of the accuracy benefit at a fraction of the token cost.

Best for: varied or unpredictable question types where writing targeted examples isn't practical; prototyping and early-stage testing; running lighter models on mid-difficulty tasks.

Template:

{{your question}}

Before giving your final answer, break down your reasoning step by step, with each step on its own line.

The small addition of "each step on its own line" consistently produces more structured output than a bare "step by step" instruction — which matters if you're parsing the response programmatically.

---

Tree of Thought (ToT)

ToT, introduced by Princeton and Google in 2023, extends the single reasoning chain into a branching search. Instead of committing to one line of reasoning, the model explores multiple candidate approaches, evaluates each, and pursues the most promising one.

You can implement this in two ways:

• Single-prompt simulation: ask the model to generate multiple candidate approaches within one response, score them, and then expand the best one
• Multi-call orchestration: use code to manage multiple conversation turns, generating one branch per call and applying external pruning logic

Best for: creative planning, architectural decisions in software, research questions with no single right answer — problems where the solution space is wide and tradeoffs matter. For math problems with one correct answer, ToT rarely outperforms standard CoT and costs significantly more.

Single-prompt template:

This is a problem that benefits from exploring multiple approaches: {{your question}}

Work through it as follows:
1. Outline 3 distinct approaches (2-3 sentences each)
2. Score each approach from 0-10 and briefly note its strengths and weaknesses
3. Expand the highest-scoring approach into a full solution
4. State your final answer

---

Choosing Between Them

	Few-Shot CoT	Zero-Shot CoT	Tree of Thought
Setup effort	High	Minimal	Medium–High
Token cost	High	Low–Medium	High
Task type	Structured, single answer	General purpose	Open-ended, multiple solutions
Model dependency	Low (examples do the work)	Medium–High	High
Implementation complexity	Low	Lowest	High

In practice, the decision tree is simple: start with zero-shot CoT to validate your approach quickly. If accuracy isn't there, move to few-shot CoT. Only reach for ToT when the problem genuinely requires exploring competing solutions.

---

Engineering Details Worth Knowing

Force a structured output format. In production, you'll probably need to parse the reasoning separately from the final answer. Adding wrap your reasoning in <thinking>...</thinking> and your final answer in <answer>...</answer> makes that extraction trivial and works reliably with most frontier models.

Know when not to add CoT. Models like DeepSeek R1 and OpenAI's o-series have CoT baked into their inference process — they already "think" internally before responding. Adding an explicit "step by step" instruction can sometimes conflict with those built-in mechanisms. Test before assuming it helps.

Temperature settings matter. For CoT tasks, keep temperature between 0 and 0.3 to reduce erratic jumps in reasoning chains. If you're using ToT and want diverse candidate approaches, pushing temperature up to 0.7 during branch generation is reasonable.

Self-verification as a final step. Models can produce reasoning chains that sound coherent but contain subtle errors — a step that skips a logical leap, or arithmetic that's slightly off. For math or code tasks, appending review each step above for logical gaps or calculation errors before the final answer acts as a lightweight self-check and catches a surprising number of mistakes.

---

At its core, chain of thought prompting is a negotiation about where the model spends its compute — pushing more tokens into intermediate reasoning rather than jumping straight to a conclusion. The templates above will get you most of the way there; the engineering notes will keep you out of the common traps.

When I'm running comparison tests across models — say, checking whether few-shot CoT on Claude Opus outperforms zero-shot on GPT-4o for a specific task type — I route everything through XycAi. One OpenAI-compatible API covers 200+ models, with GPT and Claude official models starting at 14% of list price, so iterating on prompts across multiple models doesn't require juggling separate accounts or blowing through a research budget.