Home / Guides / Step 5 Preview

Step 5 Preview code review benchmark: free, ties DeepSeek on small PRs, fails big ones

Published October 8, 2026 · route and catalog checked October 8, 2026 · applies to pgup-ai/jbot-review-action v0

Step 5 Preview is a reasonable free reviewer for small and medium pull requests, and the wrong choice for large ones. We ran it as J-Bot Review’s model on nine pull requests. On the six production changes it finished, 4 to 13 files each, it found 8 of 31 known issues and tied DeepSeek V4.1 Flash on the two where we could compare. Its median finished review took 16 minutes, and it was slower than DeepSeek on two of the three pull requests where we have a DeepSeek time. It failed both pull requests of 20 files and more.

Key results

• Step 5 Preview, a StepFun model with a 1M-token context, costs $0 on OpenCode Zen for a limited time.
• On six production pull requests of 4 to 13 files, it found 8 of 31 known issues, with a median review time of 16 minutes.
• On two production pull requests that six models reviewed, it found 4 of 12 known issues, tied with DeepSeek V4.1 Flash and MiMo V2.6 Flash.
• It failed both pull requests of 20 and 32 files because one page did not finish, so no review posted.
• On a public 9-file pull request it found the same real bug as DeepSeek V4.1 Flash, in 8.2 minutes against 4.4.

Set up Step 5 Preview code review in three steps

1. Create an OpenCode API key. Sign in at opencode.ai, then save the key as the repository secret OPENCODE_API_KEY (Settings → Secrets and variables → Actions).
2. Commit the workflow. Add the file below as .github/workflows/jbot-review.yml. J-Bot sends new models low effort by default, so the workflow sets high, the level we tested on production code.
3. Open a pull request. J-Bot reads the base…head diff and posts diff-anchored findings with a verdict. A second session checks blocking findings before they post.

name: J-Bot Code Review
on:
  pull_request: { types: [opened, reopened, ready_for_review, synchronize] }

concurrency:
  group: jbot-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write
  issues: write
  checks: read

jobs:
  review:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }
      - uses: pgup-ai/jbot-review-action@v0
        with:
          model: opencode/step-5-preview-free
          model-options: '{"reasoningEffort":"high"}'
          # Large pull requests: give each review more time than the 30-minute default.
          # time-budget-minutes: 45
          opencode-api-key: ${{ secrets.OPENCODE_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}

No J-Bot update is needed. The OpenCode build J-Bot pins, 2.0.24, ships without an entry for this model, but it reads the models.dev catalog when it starts, so the id resolved on our first run.

Step 5 Preview on OpenCode Zen: free, 1M context

Sources: the OpenCode Zen documentation (model list, pricing and privacy) and the models.dev catalog entry, both checked October 8, 2026. Free windows end, so check the pricing table before you depend on it.

Step 5 Preview vs DeepSeek V4.1 Flash on a public pull request

We started with pull request #286 in J-Bot Review’s own repository: 9 files on one review page. We ran it at low effort with verification off, the settings we had used for DeepSeek V4.1 Flash on the same change a day earlier. That DeepSeek run used an older J-Bot build (e824bd4, against 0906459 for Step 5).

Scroll horizontally to see every column.

Both models found the same real bug. A telemetry helper had gained a workspace parameter that three backend call sites did not pass. Step 5’s write-up named all three call sites. Neither model found the two known issues, shell-parsing bugs fixed later in #287. Step 5 made fewer tool calls and still took 1.9 times as long on this pull request.

Step 5 Preview benchmark on eight production pull requests

Next we ran eight pull requests from a private TypeScript monorepo. An earlier audit had written an answer key for each one, 44 known issues in total. Each review ran once, at high effort, with one review pass, verification off and J-Bot’s default 30-minute budget. A script matched findings to the key by keyword, and we read every finding it did not match by hand. We label the pull requests A to H.

Scroll horizontally to see every column.

J-Bot splits a diff into budgeted review pages and posts only when every page finishes. The raw numbers (JSON) include the DeepSeek reference runs and the cross-model comparison.

Scroll horizontally to see the whole chart.

On the six pull requests it finished, Step 5 found 8 of 31 known issues. The median review took 16 minutes.

Step 5 Preview vs DeepSeek, MiMo, GLM and other models

We also ran J-Bot with other models on two of these pull requests, A and B, with the same settings as above: high effort, one review pass and verification off. A has 7 known issues and B has 5, so each row is scored out of 12. Every model ran once, except LongCat 2.5 Preview, which ran A twice, between September 26 and October 8.

Scroll horizontally to see every column.

One run per model (two on A for LongCat 2.5 Preview), on J-Bot builds that changed between September 26 and October 8. Flash-class models swing by about two known issues between identical runs, so the top three rows are a tie, not a ranking. MiMo’s 3 on A includes one finding we judged a likely match. Runs before October 8 were scored against an earlier answer key with one more issue on B, which no model found, and every row here leaves it out.

Scroll horizontally to see the whole chart.

Step 5 Preview matched the best results in this group, DeepSeek V4.1 Flash and MiMo V2.6 Flash, and it was the only one of the three on a free route. DeepSeek was the fastest of the three on A, and Step 5 was the fastest of the three on B. GLM 5.3 Flash found half as many. Muse Spark 1.3 finished both reviews in under four minutes and found almost nothing.

Why the large pull requests failed

Each failure came from a single review page. On G, 20 files over three pages, one page returned an empty response, and its retry used up the rest of the budget. On H, 32 files over four pages, one page ended with a partial result.

J-Bot will not post a review that skips part of the diff. When a page fails after its retry, the whole run fails and nothing posts. That rule protects you from a review that looks complete and isn’t. With a slow model it also means one stuck page costs the entire review. This result decides how to use Step 5. A model that handles 13 files and stalls at 20 can’t be the default for a repository that opens large pull requests.

When to use Step 5 Preview

• Small and medium pull requests, while it’s free. Up to about 13 files it finished every review. On the two pull requests we ran across models, it found as many known issues as DeepSeek V4.1 Flash and MiMo V2.6 Flash, at no cost.
• Not for large pull requests. Both of ours failed. If you try it on changes that size anyway, raise time-budget-minutes to 45 or more. We have not yet rerun G and H with a longer budget.
• Not as a default in a model pool. A pool picks a model per pull request, so Step 5 would eventually draw a large change and fail it. Keep it as an explicit choice for repositories whose pull requests stay small.
• Not when review speed matters. Only three pull requests have a DeepSeek V4.1 Flash time, all from older J-Bot code. Step 5 was 1.9 times slower on #286 and on A, and faster on B. For faster free reviews, see the free model comparison.

What this test can’t tell you

Each pull request ran once. Flash-class models swing by about two known issues between identical runs, so a single row can mislead in either direction. Eight production pull requests from one repository are a screen, not a benchmark.

The model comparisons are thin. The cross-model table rests on two pull requests with one run per model (two on A for LongCat), on J-Bot builds from September 26 to October 8, and we did not run the other models on C through H. The public pull request ran at low effort and the production ones at high, so the two tables don’t combine.

The production repository is private, so we can’t publish those diffs or findings. The public run can be checked against #286, #287 and #293, which fixed the telemetry bug.

Where the diff goes

• To OpenCode and its model provider. The review runs read-only on your GitHub Actions runner, and J-Bot sends the diff to OpenCode Zen. OpenCode’s Zen documentation says this model’s provider follows a zero-retention policy and does not use your data for training. That is OpenCode’s commitment, not J-Bot’s.
• The key stays in the server process. Each review session’s shell gets an allowlisted environment, so a tool call inside the review can’t read OPENCODE_API_KEY.
• Fork pull requests can’t read the key. GitHub doesn’t pass repository secrets to pull_request workflows from forks.

FAQ

Related