Home / Guides / DeepSeek experiment results

Engineering notes · Experiment

DeepSeek V4.1 Flash experiment results and methodology

By PGUP AI · Published October 3, 2026

The data behind our DeepSeek V4.1 Flash code-review experiments: 36 valid repeated reviews, separate exploratory screens and one excluded slim-compliance attempt. They support comparisons within each study, not a pooled model benchmark.

Download the sanitized JSON results or read the experiment narrative. The source PRs are private. We publish anonymous case labels, review settings and measurements. Repository identities, exact diff sizes, code, finding details, prompts and internal report identifiers are omitted. You can check the arithmetic, but you can’t independently verify the findings from this export.

Study definitions and controls

All rows below use direct DeepSeek V4.1 Flash, low reasoning and at most 10 concurrent sessions. Full reviews ran sequentially, in local dry-run mode. Native includes existing context packs, compliance and the normal supplied-evidence-first verifier with repository fallback. Zero-candidate runs naturally have nothing to verify.

• Search context: three frozen cases × two arms × three repetitions. Main-only source augmentation with eight attempts per main session.
• Search recognition: fresh case C controls after fixing eligibility for searches prefixed by a literal change-directory command. Other augmentation limits remain.
• Compliance ablation: six fresh case B reviews, native versus compliance disabled. Main receives its normal scheduling notice to inspect omitted guidance when applicable. Neither arm detected the reference concern identified before the experiment.
• Slim compliance: fresh case C controls; scoped-rule removal after native pagination and code-context assembly. No search augmentation and no new lookup caps.

Scroll the table horizontally to see every column.

Elapsed times below cover the review pipeline, excluding small process startup and teardown overhead. Phases can overlap. Costs are estimated USD from saved SDK/provider usage. Provider cache state was not reset. Local shell behavior and dry-run prior-thread handling limit transfer to GitHub CI timings.

Every valid repeated run

N = native. T = main-only search context in the first two studies, slim compliance in the third, or compliance disabled in the ablation. M/C counts are raw main/compliance candidates before filtering. Published means the pipeline would publish in dry-run mode; no comments were sent, and this count is not a count of independently confirmed bugs.

Main-only search context

Scroll the table horizontally to see every column.

Corrected search recognition

Scroll the table horizontally to see every column.

Slim compliance

Scroll the table horizontally to see every column.

Compliance-off ablation · not adopted

Verification stayed enabled but had no candidates to check. The quality reference was missed in all six runs.

Scroll the table horizontally to see every column.

Exploratory pass-placement and shard screens

One observation per configuration on case C. These are separate from the repeated comparisons. The main-only screen has the earlier eight-attempt augmentation allowance; newer auxiliary and all-stage screens remove attempt, match-count and source-count caps, while byte budgets and collection deadlines remain. Do not infer a placement-only causal effect across those variants.

Scroll the table horizontally to see every column.

Augmented screens also perform controller source loads outside the model tool-call counter. The all-stage screen used 178 extra source loads and 272 model tool calls, versus 260 model tool calls for its native observation. Its 343.4-second review did not demonstrate equivalent finding coverage. The shard setting changes both main and compliance grouping, so it is a whole-pipeline comparison.

What the final slim-compliance quality counts mean

Scroll the table horizontally to see every column.

Only one supported concern was shared between arms. One assessor checked findings against source with knowledge of earlier results. The counts describe those assessments, not full recall or independent blind grades. The rule for the reference concern in the last row was present in both arms.

Excluded, historical and untested work

The original second native attempt in the slim study took 503.665 seconds and cost an estimated $0.3983. It wrote an untracked diff cache, violating the predeclared clean-fixture condition. Tracked source was unchanged. We preserved the failure, restored the fixture and retried with identical source and configuration. Six valid slim-study runs cost $2.3434; including that attempt, $2.7416. The JSON retains its reason, time and cost.

Earlier staged-hybrid runs used OpenCode Go, including a concurrency-three setup. Later gateway attempts included quota failures. A superseded direct-API pilot used one page, while the requested automatic grouping produced four. The original GitHub job also used a different auxiliary model and prior-thread behavior. Those records informed the investigation but are not numerical controls in this public matched table.

An offline contribution audit covered 24 earlier completed reviews; it added no paid runs. It found useful compliance-only concerns on the large case. A six-run skip-compliance ablation produced no findings in either arm and missed the predefined reference concern every time. It reduced cost but did not establish quality parity; we abandoned that direction in favor of keeping the pass. The slim study’s single-page case B was a byte-identical no-op in preflight and was not rerun. Dynamic compliance depth and LLM-directed effort were not tested.

Settings were fixed within each comparison. The reviewer changed between studies, so their native runs shouldn’t be pooled into one baseline.

For the interpretation, read the DeepSeek optimization case study. For the evaluation method, read how to evaluate AI code review.