AI Code Review in Your CI/CD Pipeline: Automated PR Comments

Why Manual Code Review Is Becoming a Bottleneck

A 10-person engineering team typically merges 40–80 pull requests per week. At 20 minutes of focused review per PR, that's 800–1,600 minutes of engineer attention consumed by code review alone — before you account for context-switching costs from waiting.

The problem isn't that engineers aren't thorough. It's that a large slice of review work is mechanical: checking naming conventions, catching potential null dereferences, verifying error handling paths. That's exactly the kind of work LLMs are good at. Offload it to an AI code review CI/CD pipeline, and human reviewers can focus where judgment actually matters — architecture decisions, business logic, edge cases that require deep context.

Everything below has been validated on GitHub Actions. The core logic transfers directly to GitLab CI and Bitbucket Pipelines.

---

Choosing Your Engine: Claude Code vs. Codex

Before you write a single line of pipeline config, you need to pick a model. The two most practical options right now are Anthropic's Claude Code (backed by Claude Sonnet 4.6 / Opus 4.8) and OpenAI's Codex (GPT-5.4 series).

	Claude Code	Codex (GPT-5.4)
Context window	200K tokens	128K tokens
Code understanding	Strong at cross-file, multi-module analysis	Strong at single-file completion and refactoring
Structured output	Native JSON mode	Function calling
API format	REST / OpenAI-compatible	OpenAI standard
Best for	Large PRs with changes across multiple modules	Small PRs, focused local logic review

These two aren't mutually exclusive. A practical routing strategy: if the diff exceeds 300 lines, send it to Claude Code for the larger context window; under 300 lines, use Codex for faster response and lower cost.

---

The Core Architecture: Trigger → Diff → LLM → PR Comment

The pipeline has four stages.

1. Trigger

Fire on pull_request events for both opened and synchronize types. The synchronize type catches new commits pushed to an existing PR — skip it and you'll miss incremental changes.

on:
  pull_request:
    types: [opened, synchronize]

2. Extract the Diff

Only send the PR's changes to the model — not the entire repository. Use git diff against the base branch:

git fetch origin ${{ github.base_ref }}
git diff origin/${{ github.base_ref }}...HEAD \
  --unified=5 \
  -- '*.ts' '*.py' '*.go' '*.java' \
  > pr_diff.txt

--unified=5 keeps five lines of context on each side, which is enough for the LLM to understand intent. The glob filter restricts review to source files and excludes lockfiles, markdown, and build artifacts.

3. Call the LLM

Using Claude Code with OpenAI-compatible format (which makes it easy to swap models later):

import os, json, httpx

diff = open("pr_diff.txt").read()[:60000]  # token budget cap

payload = {
    "model": "claude-sonnet-4-6",
    "messages": [
        {
            "role": "system",
            "content": (
                "You are a strict but constructive code reviewer. "
                "Output only a JSON array. Each element must have four fields: "
                "file, line, severity (info/warning/error), and comment. "
                "No explanatory text. JSON only."
            )
        },
        {"role": "user", "content": f"Review this diff:\n\n{diff}"}
    ],
    "temperature": 0.2,
    "response_format": {"type": "json_object"}
}

resp = httpx.post(
    os.environ["LLM_API_BASE"] + "/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}"},
    json=payload,
    timeout=60
)
comments = json.loads(resp.json()["choices"][0]["message"]["content"])

temperature=0.2 is the key setting. Too high and the model hallucinates problems that don't exist; too low and it misses subtle bugs. Testing consistently shows 0.2–0.3 gives the best balance of precision and recall.

4. Post PR Comments

Use the GitHub REST API to create inline review comments, each tied to a specific file and line number:

import requests

gh_headers = {
    "Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}",
    "Accept": "application/vnd.github+json"
}
repo = os.environ["GITHUB_REPOSITORY"]
pr_number = os.environ["PR_NUMBER"]
commit_sha = os.environ["GITHUB_SHA"]

for item in comments:
    if item.get("severity") in ("warning", "error"):
        requests.post(
            f"https://api.github.com/repos/{repo}/pulls/{pr_number}/comments",
            headers=gh_headers,
            json={
                "body": f"**[{item['severity'].upper()}]** {item['comment']}",
                "commit_id": commit_sha,
                "path": item["file"],
                "line": item["line"]
            }
        )

Post warning and error findings as inline comments. Collect info-level notes into a single PR summary comment instead — otherwise the thread fills up with noise on every push.

---

Engineering Details Worth Getting Right

Token budget. Cap single-review input at 60K tokens. For very large PRs, batch by file, run separate LLM calls per batch, then merge results.

Idempotency. Every synchronize trigger should delete the bot's previous comments before posting new ones. A PR that gets revised five times will otherwise accumulate five rounds of stale feedback. Cleanup command:

gh api repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER/comments \
  --jq '.[] | select(.user.login=="github-actions[bot]") | .id' \
  | xargs -I{} gh api -X DELETE repos/$GITHUB_REPOSITORY/pulls/comments/{}

False positive rate. Add an explicit instruction in the system prompt: "Only report issues you are confident about. Stay silent on stylistic preferences you are uncertain about." This alone drops false positive rates from roughly 30% to under 10%.

Secrets hygiene. Store LLM_API_KEY in GitHub Secrets, never hardcoded in the workflow file. Don't echo it in any step — GitHub Actions masks secret values automatically, but an explicit echo can still leak them in certain log contexts.

Cost reality check. A 200-line diff reviewed with Claude Sonnet 4.6 consumes roughly 3K–5K tokens — under $0.01 per review at standard API pricing. For a 10-person team, total monthly LLM spend on PR review typically lands between $3–$7. Compared to the engineering time saved, it's rounding error.

---

Fitting AI Review Into Your Review Culture

The real value of an AI code review CI/CD integration isn't replacing human reviewers — it's raising the floor before they ever open the PR. The bot completes its first pass within 90 seconds of PR creation. By the time a human reviewer arrives, there's already an annotated pre-review highlighting potential issues. That changes the cognitive load of review, not the act of reviewing itself.

One team norm worth establishing: any issue the bot flags as error must be either resolved or explicitly acknowledged (with a reason) before the author requests human review. This preserves engineer judgment as the final word while eliminating the "merge first, fix later" habit.

---

For the API layer in my own setup, I use XycAi. It's OpenAI-compatible, so the Python code above works across Claude Code, Codex, and Gemini CLI by changing a single environment variable — no rewrites to business logic. For CI/CD workloads with high call volume, the pricing difference is significant: Claude Sonnet 4.6 and GPT-5.4 start at 14% of list price. Claude Code and Codex CLI are also supported with one-command setup, which saves the hassle of configuring your own proxy.