AI Code Security Review: Automate Vulnerability Detection in PRs

Why Human Reviewers Miss Security Bugs

A mid-sized engineering team merges 30–80 pull requests a day. The average reviewer spends under eight minutes per PR. SQL injection, hardcoded secrets, unsafe deserialization — engineers know what these look like. The problem isn't knowledge, it's attention. Eight minutes of focused review is hard enough once. Doing it fifty times in a row is a different matter entirely.

Veracode's data puts a number on this: over 70% of critical security vulnerabilities are already present when code hits the main branch. The review process simply didn't catch them. Traditional static analysis tools like SonarQube and Semgrep are good at rule matching, but they struggle with cross-function logic flaws and implicit code smells. Their context window is narrow and their false positive rate climbs quickly on anything that isn't a textbook pattern.

AI code security review fills that gap. It doesn't replace static scanning — it adds a semantic layer on top of the rule engine. The model can reason about variable naming intent, follow call chains, and spot contradictions between comments and implementation. This article turns that capability into a production-ready automated pipeline.

---

Designing a Security-Focused Prompt

Prompt quality determines output quality. A generic "review this code" prompt produces generic results. You need a structured template that makes model output predictable and machine-parseable every time.

Here's a four-section structure that works well in practice:

### ROLE
You are a senior application security engineer.
Your job is to review the following code diff for:
1. Security vulnerabilities (OWASP Top 10, CWE Top 25)
2. Code smells that increase attack surface
3. Secrets or credentials in plaintext
4. Unsafe dependencies or import patterns

### CONTEXT
Language: {language}
Framework: {framework}
Business logic summary: {summary}

### DIFF
{git_diff}

### OUTPUT FORMAT
Return a JSON array. Each item:
{
  "severity": "critical|high|medium|low|info",
  "category": "security|smell|secret|dependency",
  "line": <line number or null>,
  "finding": "<concise description>",
  "recommendation": "<actionable fix>"
}
If no issues found, return [].

A few deliberate design choices worth explaining:

Give the model a specific identity. Telling the model it's an AppSec engineer, not a general assistant, shifts its output toward security analysis rather than style suggestions. It's a small change that consistently improves signal quality.

Constrain the category field. Downstream scripts can filter by category — route critical + security findings to a Slack alert, everything else to a PR comment. Without structured categories, you're stuck parsing free text.

Require JSON output. Free-text responses can't be processed programmatically. JSON is the baseline format for anything a machine needs to consume.

Send the diff, not the full file. Passing only changed lines cuts token consumption by 60–80% and keeps model attention focused on what actually changed.

---

Wiring It Into GitHub Actions

The goal is a CI job that triggers on every PR, runs the AI review, and posts results directly as a PR comment.

Install dependencies:

pip install openai PyGithub

Core script — ai_review.py:

import os, json, sys
from openai import OpenAI
from github import Github

client = OpenAI(
    api_key=os.environ["AI_API_KEY"],
    base_url=os.environ.get("AI_BASE_URL", "https://api.openai.com/v1")
)

PROMPT_TEMPLATE = """..."""  # the four-section template from above

def get_pr_diff(repo_name: str, pr_number: int) -> str:
    gh = Github(os.environ["GITHUB_TOKEN"])
    repo = gh.get_repo(repo_name)
    pr = repo.get_pull(pr_number)
    diff_parts = []
    for f in pr.get_files():
        if f.filename.endswith(('.py', '.js', '.ts', '.go', '.java', '.php')):
            diff_parts.append(f"### {f.filename}\n{f.patch or ''}")
    return "\n\n".join(diff_parts)

def analyze(diff: str, language: str = "mixed") -> list:
    prompt = PROMPT_TEMPLATE.format(
        language=language, framework="auto-detect",
        summary="PR diff from CI pipeline", git_diff=diff
    )
    resp = client.chat.completions.create(
        model=os.environ.get("AI_MODEL", "gpt-4o-mini"),
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1,  # low temperature for consistent security output
        response_format={"type": "json_object"}
    )
    raw = json.loads(resp.choices[0].message.content)
    return raw.get("findings", raw) if isinstance(raw, dict) else raw

def post_comment(repo_name: str, pr_number: int, findings: list):
    if not findings:
        return
    gh = Github(os.environ["GITHUB_TOKEN"])
    pr = gh.get_repo(repo_name).get_pull(pr_number)
    severity_emoji = {"critical": "🔴", "high": "🟠", "medium": "🟡", "low": "🔵", "info": "⚪"}
    lines = ["## 🔍 AI Security Review\n"]
    order = ["critical", "high", "medium", "low", "info"]
    for f in sorted(findings, key=lambda x: order.index(x["severity"])):
        emoji = severity_emoji.get(f["severity"], "⚪")
        lines.append(f"{emoji} **[{f['severity'].upper()}]** `{f['category']}` — {f['finding']}")
        lines.append(f"  > Recommendation: {f['recommendation']}\n")
    pr.create_issue_comment("\n".join(lines))

if __name__ == "__main__":
    repo, pr_num = sys.argv[1], int(sys.argv[2])
    findings = analyze(get_pr_diff(repo, pr_num))
    post_comment(repo, pr_num, findings)
    # non-zero exit blocks merge if critical findings exist
    if any(f["severity"] == "critical" for f in findings):
        sys.exit(1)

GitHub Actions workflow:

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  ai-review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install openai PyGithub
      - run: python ai_review.py ${{ github.repository }} ${{ github.event.pull_request.number }}
        env:
          AI_API_KEY: ${{ secrets.AI_API_KEY }}
          AI_BASE_URL: ${{ secrets.AI_BASE_URL }}
          AI_MODEL: "claude-sonnet-4-5"  # or gpt-4o-mini for cost-sensitive pipelines
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Setting AI_BASE_URL to any OpenAI-compatible endpoint means switching models is a one-line environment variable change, no code changes required.

---

Model Selection for Security Review

Performance varies meaningfully across models on security-focused tasks. Here's how they compare against the same test diff containing 12 known vulnerabilities:

Model	Detection rate	False positive tendency	Cost per run	Best fit
Claude Opus 4	Highest — strong contextual reasoning	Low	High	Core modules, fintech/healthcare code
Claude Sonnet 4.5	High — best cost/quality ratio	Low–medium	Medium	Day-to-day PR baseline
GPT-4.1	High — excellent code comprehension	Medium	High	Complex multi-language monorepos
GPT-4o-mini	Moderate — fast turnaround	Medium–high	Low	Draft PRs, quick early feedback
DeepSeek V3	Medium-high	Medium	Very low	Cost-sensitive teams

The practical approach: run Sonnet 4.5 or GPT-4o-mini on every PR for speed and cost efficiency, then escalate PRs tagged security-review-required to Opus 4 or GPT-4.1 for deeper analysis. Two tiers, one pipeline.

---

Reducing False Positives

The most common complaint teams have about AI code review is that false positive rates are high enough that reviewers start ignoring results entirely. A few techniques that measurably help:

Add codebase context to the prompt. Include a short security configuration note in the CONTEXT section — something like "this project uses a parameterized query ORM, raw SQL concatenation is not used." That single addition can cut related false positives by 40% or more.

Layer your alerting by severity. Treat critical and high as required checks that block merge. Let medium and low post as comments without blocking. Collapse or drop info entirely. Giving every severity level equal weight just creates noise, and noise trains reviewers to tune everything out.

Persist results and track trends. Store scan output in a database — SQLite is fine to start — and aggregate by repository, author, and vulnerability type. If the same class of issue keeps appearing, that's a signal for engineering-side remediation or a security training session, not just more AI reminders.

Run a calibration loop. Have a security engineer spot-check 20 AI findings every two weeks and annotate false positives and misses. Feed representative false positive examples back into the prompt as negative examples. Model behavior improves over time when you close the loop.

---

After running this setup across several projects, the outcome that stood out most wasn't just the drop in critical vulnerabilities reaching main — it was how reviewers' time shifted. Instead of hunting for problems, they were evaluating whether an AI finding was worth acting on. That's a more useful division of labor.

For the AI_BASE_URL in the workflow above, I use XycAi. It's a single OpenAI-compatible endpoint that routes to 200+ models — Claude Sonnet, GPT-4.1, and others — at a fraction of list price, which makes the two-tier model strategy genuinely affordable at scale. Setup for Claude Code or Codex CLI is a single command, so the same API key covers your local dev tooling and your CI pipeline without any extra configuration.