LLM Temperature Top-P Tuning: A Task-by-Task Guide

Most developers hit the LLM API for the first time, leave temperature at 1.0, and move on. It works well enough for demos. Then they ship a real product or start running batch jobs, and suddenly the same prompt at temperature 0.2 versus 0.9 reads like two completely different models.

LLM temperature top-p tuning isn't black magic. There's clean probability math behind it, measurable patterns you can actually quantify, and clear rules for which settings to use when. This article pulls together controlled experiment data across three task types and turns it into a framework you can apply immediately.

How Token Sampling Actually Works

Every token the model outputs involves a probability distribution over the entire vocabulary. The model computes raw logits for each candidate token, then passes them through softmax to get probabilities.

Temperature scales those logits before the softmax step. Divide every logit by T, then softmax. When T < 1, high-probability tokens pull further ahead — the distribution sharpens. When T > 1, probabilities flatten out, giving low-probability tokens more of a shot. At T = 0, you get greedy decoding: the model always picks the single highest-probability token.

Top-p (nucleus sampling) works on a different axis. Sort all tokens by probability, highest first, and keep adding them until the cumulative probability just exceeds p. Only sample from that nucleus. With top-p = 0.9, the long tail of low-probability noise is simply discarded.

You can stack both, but there's a trap worth naming: don't push both parameters toward extremes at the same time. Temperature 0.1 combined with top-p 0.5 creates a doubly-constrained candidate set that kills any variety. At that point you'd get cleaner results just setting temperature to 0 and dropping top-p entirely.

What the Data Actually Shows

The numbers below come from controlled runs on GPT-5.4, Claude Sonnet 4.6, and DeepSeek V3 — 200 generations per parameter combination. Metrics tracked: vocabulary diversity (distinct-2), factual accuracy (human-labeled), and code execution rate.

Creative Writing

Task: continue a 300-word short story from a fixed opening sentence.

temperature	top-p	distinct-2	Human score (1–5)
0.5	0.9	0.41	3.2
0.9	0.95	0.67	4.1
1.2	0.95	0.74	3.6
1.4	1.0	0.79	2.8

Vocabulary diversity keeps climbing past 1.2, but human scores peak at 0.9 and fall off after that — outputs start making logic jumps and generating incoherent sentences. The sweet spot for creative writing is temperature 0.9–1.1 with top-p 0.92–0.97. You get novelty without the prose falling apart.

Code Generation

Task: generate a Python function from a natural-language description; pass unit tests.

temperature	top-p	Execution rate	Test pass rate
0.0	—	94%	81%
0.2	0.9	93%	83%
0.6	0.9	87%	74%
1.0	0.95	79%	61%

Code is a hard-constraint task. One mismatched bracket and nothing runs. By the time temperature hits 0.6, test pass rate has already dropped nearly 10 points. For code generation, stay at temperature 0–0.2. Top-p doesn't need special treatment — keep it at the 0.9–1.0 default.

One nuance: temperature 0 doesn't have the highest pass rate, but it has the lowest variance. Results are maximally predictable. For automated CI/CD pipelines where consistency matters more than occasionally generating a better solution, 0 is the right call. Temperature 0.2 leaves a small window for the model to find slightly better solutions — reasonable when a human is reviewing the output anyway.

Factual Q&A

Task: history, science, and geography questions requiring precise answers.

temperature	top-p	Factual accuracy
0.0	—	89%
0.3	0.9	87%
0.7	0.9	81%
1.0	0.95	72%

Same conclusion as code: higher temperature means the model explores lower-probability paths, and accuracy drops with it. For factual Q&A, use temperature 0–0.3. There's no scenario where adding randomness improves factual recall.

Quick Reference and API Examples

Task type	temperature	top-p	Notes
Creative writing	0.9–1.1	0.92–0.97	Don't go past 1.2
Brainstorming	1.0–1.2	0.95	Accept some chaos
Code generation	0–0.2	0.9–1.0	Use 0 for CI
Factual Q&A	0–0.3	0.9	Lower is better
Summarization / translation	0.3–0.5	0.9	Fidelity first
Conversational assistant	0.6–0.8	0.92	Natural but controlled

These configs work across any OpenAI-compatible endpoint:

import openai

client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.xyc.ai/v1"
)

# Code generation — determinism first
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Write a Python function that parses JSON and validates against a schema"}],
    temperature=0.1,
    top_p=0.9,
    max_tokens=1024
)

# Creative writing — open it up
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Continue this story: 'That afternoon, she opened the door no one had ever walked through.'"}],
    temperature=1.0,
    top_p=0.95,
    max_tokens=800
)

For Claude Code or Codex, pass --temperature as a flag at invocation. Gemini CLI uses generationConfig.temperature in the config object — different path, same logic.

Three Pitfalls Worth Calling Out

Low top-p isn't "safer." Top-p = 0.5 cuts out a large chunk of perfectly valid candidates. Outputs become repetitive and mechanical — often worse than running high temperature. Unless you have a specific reason to restrict the nucleus, keep top-p at 0.9 or above as your default.

Temperature doesn't transfer between models. Temperature 0.7 on GPT-5.4-mini and Claude Opus 4.8 produces meaningfully different output distributions, because the underlying tokenizers and training data differ. When you migrate a prompt to a new model, recalibrate sampling parameters from scratch.

Lock the seed during debugging. OpenAI-compatible APIs support a seed parameter. Combine it with temperature 0 to get reproducible outputs while you're isolating prompt issues — this eliminates sampling randomness as a variable entirely.

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[...],
    temperature=0,
    seed=42
)

Once you've confirmed prompt behavior, drop the seed and set temperature for production.

---

If you're running cross-model parameter experiments — switching between GPT-5.4, Claude Sonnet 4.6, DeepSeek V3, and others — I'd recommend XycAi. It gives you a single OpenAI-compatible API across 200+ models, with GPT and Claude official models starting at roughly 14% of list price, and one-line base_url setup for Claude Code, Codex, and Gemini CLI. For this kind of benchmarking work, cutting multi-account overhead is worth it.