OpenCodeReview Review: What Alibaba's Open-Source AI Code Reviewer Gets Right, and What We Found Testing It

OpenCodeReview is Alibaba's open-source AI code reviewer, and its real contribution is an architecture, not a model or a score. It decides in ordinary code which files get reviewed, which rules apply to each file and where each comment lands. It hands the language model only the judgment call: is this code wrong? On its own benchmark that makes it far quieter and about nine times cheaper than running Claude Code as a reviewer with the same model, at the cost of finding fewer real bugs. We read the Go source, redid the benchmark arithmetic and ran the tool on a test repository. Our verdict: adopt the design, pin the version and model, write your own security rules for Java, and do not read the benchmark as "finds more bugs". It finds fewer bugs and wastes far less of your reviewers' time, which for most teams is the right trade.

What OpenCodeReview is, in one minute

OCR (the CLI is ocr) was Alibaba's internal AI review assistant before it was released under Apache-2.0. The README says it served tens of thousands of developers over two years; we have no way to check that, so treat it as the company's claim. As of 21 October 2026 the repository has about 47,800 GitHub stars and the npm package is at version 1.13.2.

It does four things:

• Diff review (ocr review) of workspace changes, a branch range or one commit, with resumable sessions.
• Full-file scans (ocr scan) of a directory with no diff, for auditing code nobody has reviewed.
• Delegation mode (ocr delegate), where OCR does file selection and rule resolution and your own coding agent (Claude Code, Codex, Cursor, Kimi Code, OpenCode) does the reviewing with its model. No OCR API key is needed.
• CI integration for GitHub Actions, GitLab CI, GitFlic and Gerrit, with an OpenTelemetry hook and a local viewer for review sessions.

It needs Git 2.41 or later and any OpenAI- or Anthropic-compatible endpoint.

How it works: code does the bookkeeping, the model does the judging

The README's diagnosis of general-purpose agents used as reviewers matches what we see in customer deployments. On large change sets they skip files, their line numbers drift off target, and their quality swings with small prompt changes. OCR's answer is to take every step that must not go wrong away from the model.

Source: OpenCodeReview implementation. The diagram separates deterministic code from model calls.

Four details from the source are worth copying into any agent you build.

Selection is a pure function. selectFiles applies path and extension gates, skips deletions and enforces a per-file token ceiling. It has no LLM and no Git access, so the --preview command and the real run compute the same answer. A comment in the code says the two used to drift apart. If your agent's dry run and real run can disagree, you do not have a dry run.

Grouping uses the model, cheaply. The README calls bundling "deterministic engineering", but grouping.go asks the LLM to group files using metadata only (status, path and lines added and removed, no diff content), returning integer indexes rather than paths to save output tokens. Change sets under four files skip the call: they are reviewed together if their total churn is under 200 lines, otherwise one file per sub-agent. Any error falls back to one file per group. That is a sensible design. It is not purely deterministic, and the distinction matters when you debug why two files were reviewed apart.

Positions come from quotes, never from the model. Every comment carries the existing code it refers to, and OCR finds that text in the diff to compute line numbers. Only if the text match fails does a second model call extract the exact snippet, after which matching runs again. This is the single most transferable idea in the repository: never ask a language model for a line number. Ask it for the words, and let code count.

The filter is told which mistake is worse. The review-filter prompt says that keeping a wrong comment costs a reviewer a few seconds, while removing a correct one destroys a finding nobody will ever see. It may only drop a comment the diff proves wrong, and "I cannot verify this" means keep. That asymmetry, written in plain words, is why the filter helps precision without gutting recall.

The benchmark, read properly

OCR's README shows a leaderboard on AACR-Bench: 50 open-source repositories, 200 real pull requests, 10 languages, 1,505 verified issues. We recomputed what each row means for a team reading the comments. The chart below divides by 200 pull requests.

Sources: repository benchmark and AACR-Bench paper. The chart uses our per-PR arithmetic on the reported results.

Five things stand out once you do the division.

The headline is noise, not bugs found. With Claude 4.6 Opus, Claude Code found 435 of the 1,505 issues and OCR found 301. Claude Code had higher recall in every one of the five same-model pairs. What OCR wins is volume: about 3 unmatched comments per PR against 28. A reviewer facing thirty comments a pull request stops reading by Wednesday.

"About 1/9 of the tokens" checks out. Pooled across the five same-model pairs, Claude Code used 9.0 times the tokens of OCR (the ratio ranges from 5.4 to 14.7 by model).

Absolute quality is low for everyone. The best F1 in the table is 25.1%. Even the quietest configuration has about one comment in three matching a verified issue. AI review in late 2026 is a first pass, not a reviewer.

The benchmark is not independent. AACR-Bench (arXiv 2601.19494) is co-authored by Nanjing University and Alibaba. Its candidate issues were generated by "an internal review system" and by Claude Code, then verified by 80 engineers, and 1,114 of the 1,505 issues came from that model-plus-expert augmentation. A comment counts as correct when an LLM matcher (Qwen3-235B) judges it equivalent to a verified issue with overlapping lines. That is a reasonable method, and it means precision is a floor: a real bug nobody in the pipeline found is scored as a false alarm. It also means a reviewer whose style resembles the internal system has a home-ground advantage. We would like to see the same table on a benchmark Alibaba did not help build.

A newer model scored worse. Claude 4.8 Opus under OCR 1.3.1 scored 17.9 F1 against 25.1 for 4.6, with recall nearly halved. The row mixes versions too: Qwen3.8-Max ran on OCR 1.8.7 while the others ran on 1.3.1. A review harness is tuned to the habits of the model it was built with. Treat a model upgrade as a change to your review system and rerun your own sample first.

What we found when we ran it

We downloaded the 1.13.2 Linux release, checked it against the published SHA-256, and built a small repository with one realistic feature commit: a Java service change that adds a stock check and a string-concatenated SQL query, an English message added without its Chinese counterpart, a new lodash dependency, an innerHTML assignment in JavaScript, a workflow switched to pull_request_target that checks out the PR head, a deleted file, a 3,000-package lockfile and a 60,000-line generated JavaScript file. Then we ran ocr delegate preview and ocr delegate rule, which show exactly what the model would be given without calling one.

The Java finding generalizes. The rule files for Python, PHP and Rust tell the model to flag SQL built from strings, and the MyBatis mapper XML and default rules name SQL injection directly. Java files get java.md, which has no security section at all, and neither do the Kotlin and C++ files (the C file at least covers buffer overflows). The default rules, which do ask about SQL injection and XSS, apply only to file types with no rule file of their own, so they never reach a Java file. The model may still flag the injection on its own judgment; what the rule table guarantees is that it will not be prompted to look. The repository description promises a ruleset with "NPE, thread-safety, XSS, SQL injection", which is true across the rule set and not true for a Java file.

This is the cost of OCR's design, stated fairly. Rules that focus the model's attention also decide what it does not look at. Precision and coverage are bought from the same budget, and you should know where your tool spent it.

Pros and cons

Where it fits, and where it doesn't

Good fits. Teams with a lot of pull requests and few senior reviewers, where the bottleneck is attention. Platform teams that want one review policy across many languages with per-path rules. Agentic coding setups where agents open many PRs and a human must triage them; we argued in our agentic software factory guide that review becomes the constraint, and a quiet first-pass reviewer helps there. And for forward deployed engineers, ocr scan is a fast way into an unfamiliar customer codebase: run it over the modules you are about to touch and read the findings as a map of where the risk sits.

Poor fits. Security review. Regulated changes where a missed defect is expensive. Small teams with a handful of PRs a week, where a general agent's extra recall costs little attention. And any setup where nobody will maintain the rule files, because the rules are the product.

How we would roll it out

1. Shadow for two weeks. Run OCR on every PR but post to a log, not the PR. Have reviewers tag a sample of comments as useful, wrong or trivial. That gives you your own precision, which is the only number that matters.
2. Write the rules you are missing. For Java, add injection, deserialization, path traversal and secrets checks to a custom rule file. Add ignore rules for generated code, vendored code and snapshots.
3. Pin the version and the model. Rerun your tagged sample before changing either. The published table shows a model upgrade can halve recall.
4. Keep a security scanner and a dependency scanner in CI. OCR excludes lockfiles and is not a SAST tool.
5. Measure what gets acted on. Track the share of comments that lead to a code change. If it falls below about one in four, people have stopped reading and the bot is a cost.
6. Pair it with a general agent on a schedule for the risky directories, filing issues rather than comments.

The cost is easy to estimate. The benchmark's OCR runs used 334K to 743K tokens per review. At an illustrative 3 dollars per million input tokens, 400K tokens is about 1.20 dollars a pull request before caching, and a team merging 500 PRs a month spends about 600 dollars. The reviewer hours saved by not reading 25 extra comments a PR are worth more than that by an order of magnitude.

What to take from it, even if you never install it

OpenCodeReview is the clearest public example of a rule we apply to every agent we put in front of a customer: give the model the judgment, give code the bookkeeping. File coverage, positions, rule selection and resumability are bookkeeping. They should be code you can test, preview and diff. That is the same lesson as harness engineering, and the same reason how coding agents find code matters as much as which model you pick.

If you interview for an applied AI or FDE role, expect a version of this as a design question: "Design an AI code reviewer people don't mute." The strong answer separates deterministic steps from model steps, measures precision on the team's own PRs, and talks about reviewer attention as the scarce resource. Practice that kind of answer in our system design questions and RAG and agent questions, and read SWE-bench for the benchmark-reading habits this post relied on.