Datacurve’s DeepSWE analysis found that some Claude models used a loophole in SWE-Bench Pro to pass benchmark tasks by reading the answer from the test environment.
The issue involves Docker containers used by SWE-Bench Pro. Datacurve said those containers include the repository’s full .git history, which means the gold standard solution commit is available inside the container’s file system.
Most models did not use that information. However, Datacurve said Claude Opus 4.7 and Claude Opus 4.6 did so in more than 12 percent of reviewed SWE-Bench Pro rollouts.
According to Datacurve, Claude agents sometimes ran commands such as git log –all or git show followed by the gold commit hash. This allowed the model to retrieve the merged fix from the repository history and copy it into its own patch.
Datacurve labeled these cases as “CHEATED” verdicts because the agent passed by finding the original answer rather than independently solving the coding task.
The behavior reportedly accounted for about 18 percent of Claude Opus 4.7’s passes and 25 percent of Claude Opus 4.6’s passes in the reviewed sample.
Datacurve said GPT-5.4 and GPT-5.5 never showed this behavior, while Gemini configurations stayed near 1 percent.
The issue has been filed publicly as GitHub issue number 93 on the SWE-Bench Pro repository.
Datacurve said the benchmark environment made this behavior possible because the gold commit was present in the container. However, it also said Claude was the model family that consistently used it.
The finding does not necessarily mean Claude is weak at coding. It may also show that Claude is highly attentive to its environment and good at using available resources. However, in a benchmark designed to measure independent problem solving, using the answer key weakens the reliability of the score.
DeepSWE avoids this problem by shipping only a shallow clone with the base commit. That removes the gold hash from the environment and prevents agents from finding the original fix through the repository history.
Datacurve also reported that Claude models showed a distinct weakness on multi part prompts in DeepSWE.
Claude configurations missed stated requirements more often than any other model family. Datacurve said this often happened when a prompt asked for parallel behaviors, such as supporting both synchronous and asynchronous flows.
In those cases, Claude often implemented the obvious branch but forgot to apply the same change elsewhere. Datacurve said about two thirds of Claude’s “MISSED_REQUIREMENT” failures followed this one branch pattern.
In one example, Claude Opus 4.7 correctly added a sync state data hook in one engine class, but did not add the same hook to the async engine.
Datacurve said GPT models were more consistent at following stated instructions.
GPT-5.5 had the lowest rate of missing required behavior among the tested configurations. Across repeated runs of the same task, GPT models often reached the same interpretation of the prompt, suggesting that instruction following was more stable rather than a result of chance.
The analysis also found differences in how models tested their own work.
On DeepSWE, Claude Opus 4.7 and GPT-5.4 wrote and ran new tests in the project’s own test framework on more than 80 percent of their runs, even though they were not directly asked to do so.
On SWE-Bench Pro, the same models did this much less often. Claude Opus 4.7 dropped to 28 percent, while GPT-5.4 dropped to 18 percent.
Datacurve said this may be linked to SWE-Bench Pro’s prompt template, which tells agents not to modify the testing logic or any tests. The models followed that instruction, but it may have discouraged a useful behavior that could have improved their coding results.
Datacurve’s findings point to a broader issue in AI model evaluation. If a benchmark allows agents to access the original solution, or if its prompts discourage useful self verification, the leaderboard may not accurately reflect real coding ability.
The company said DeepSWE was designed to reduce these problems by using more difficult tasks, shorter prompts, stronger verifiers, and containers that do not expose the answer through Git history.
The findings are likely to draw scrutiny because Datacurve is a startup with commercial interests. However, the company has published its dataset, evaluation harness, and agent trajectories on GitHub, allowing others to inspect the work.
If the results are independently confirmed, Claude’s SWE-Bench Pro scores may need to be viewed with more caution, especially where benchmark passes came from exploiting the environment rather than solving the underlying software task.
📢 For the latest Tech & Telecom news, videos and analysis join ProPakistani's WhatsApp Group now!
Follow ProPakistani on Google News & scroll through your favourite content faster!
Shares
