![\"Comparison](\"https:\/\/www.newsbeep.com\/us\/wp-content\/uploads\/2026\/05\/1-Overall-AI-Capability.png\")
<\/p>\nIn April 2026, the Center for AI Standards and Innovation (CAISI) evaluated the open-weight AI model DeepSeek V4 Pro (\u201cDeepSeek V4\u201d). CAISI evaluations indicate that DeepSeek V4\u2019s capabilities lag behind the frontier by about 8 months (Figure 1).<\/p>\n
<\/p>\n
Figure 1: Comparison of aggregate capabilities over time of the most capable publicly released U.S. and PRC models according to a suite of benchmarks covering five domains.
Every 200-point increase on the y-axis equates to a 3x increase in the odds of solving a given task. Model capability was fitted using an approach inspired by Item Response Theory (IRT), as detailed in the Appendix. 16 benchmarks across 35 models were used to produce this figure. Trend lines were fit with least squares regression on frontier models. Error bars denote 95% CIs.<\/p>\n
\u00a0<\/p>\n
Key FindingsDeepSeek V4 is the most capable PRC AI model evaluated by CAISI to date. CAISI evaluations span the domains of cyber, software engineering, natural sciences, abstract reasoning, and mathematics (Figure 2).DeepSeek V4 scores better on DeepSeek\u2019s self-reported evaluations than on CAISI evaluations. According to DeepSeek\u2019s data, DeepSeek V4 is about as capable as Opus 4.6 and GPT-5.4, which were released about 2 months ago. However, CAISI\u2019s evaluations, which include non-public benchmarks, indicate that DeepSeek V4 performs similarly to GPT-5, which was released about 8 months ago (Figure 3).DeepSeek V4 is more cost efficient than other models of similar capability. Compared to the most cost-competitive U.S. reference model (GPT-5.4 mini), DeepSeek V4 was more cost efficient on 5 out of 7 benchmarks. On the 7 benchmarks, DeepSeek V4 ranged from 53% less expensive to 41% more expensive.<\/p>\n
\u00a0<\/p>\n
Capability ResultsDeepSeek V4\u2019s capability lags behind leading U.S. models by about 8 months (Figure 1). The methodology applied can be found in the Appendix.DeepSeek V4 is the most capable PRC model to date across the domains that CAISI evaluated: cyber, software engineering, natural sciences, abstract reasoning, and mathematics. CAISI evaluated models on nine benchmarks across these five domains, including two held-out and uncontaminated benchmarks: ARC-AGI-2\u2019s semi-private dataset, and CAISI\u2019s internally-built software engineering evaluation PortBench.<\/p>\n
Domain<\/p>\n
Benchmark<\/p>\n
Model (reasoning level)<\/p>\n
OpenAI GPT-5.5
(xhigh)<\/p>\n
OpenAI GPT-5.4 mini
(xhigh)<\/p>\n
Anthropic Opus 4.6
(max)<\/p>\n
DeepSeek V4 Pro
(max)<\/p>\n
Cyber<\/p>\n
CTF-Archive-Diamond<\/p>\n
71%<\/p>\n
32%<\/p>\n
46%<\/p>\n
32%***<\/p>\n
Software Engineering<\/p>\n
SWE-Bench Verified*<\/p>\n
81%<\/p>\n
73%<\/p>\n
79%<\/p>\n
74%<\/p>\n
PortBench<\/p>\n
78%<\/p>\n
41%<\/p>\n
60%<\/p>\n
44%<\/p>\n
Natural Sciences<\/p>\n
FrontierScience<\/p>\n
79%<\/p>\n
74%<\/p>\n
72%<\/p>\n
74%<\/p>\n
GPQA-Diamond<\/p>\n
96%<\/p>\n
87%<\/p>\n
91%<\/p>\n
90%<\/p>\n
Abstract Reasoning<\/p>\n
ARC-AGI-2 semi-private**<\/p>\n
79%<\/p>\n
\u2013<\/p>\n
63%<\/p>\n
46%<\/p>\n
Mathematics<\/p>\n
OTIS-AIME-2025<\/p>\n
100%<\/p>\n
90%<\/p>\n
92%<\/p>\n
97%<\/p>\n
PUMaC 2024<\/p>\n
96%<\/p>\n
93%<\/p>\n
95%<\/p>\n
96%<\/p>\n
SMT 2025<\/p>\n
99%<\/p>\n
92%<\/p>\n
94%<\/p>\n
96%<\/p>\n
IRT-Estimated Elo<\/p>\n
1260 \u00b1 28<\/p>\n
749 \u00b1 46<\/p>\n
999 \u00b1 27<\/p>\n
800 \u00b1 28<\/p>\n
Figure 2: Summary of model performance per capability benchmark (higher is better).\u00a0
Results show accuracy (percentage of tasks solved) on each benchmark. For each benchmark, the top-performing model is highlighted and bolded. IRT-estimated Elo uncertainties reflect a 95% confidence interval. *CAISI scores on SWE-Bench Verified tend to be lower than those of other evaluators, likely due to system prompt, scaffolding, and token budget differences. **CAISI reports mean score across tasks, which differs from ARC-AGI-2\u2019s official score aggregation methodology. ***Imputed from a subset of samples via IRT.<\/p>\n
\u00a0<\/p>\n
Benchmarks<\/p>\n
This evaluation used the following benchmarks:<\/p>\n
ARC-AGI-2<\/a> Semi-Private: a non-public dataset measuring abstract reasoning from the ARC Prize Foundation<\/a>. “Semi-Private” means these tasks may have been exposed to limited third-parties.CTF-Archive-Diamond: a CAISI-developed benchmark based on 285 difficult CTF challenges drawn from the pwn.challenge cybersecurity platform developed by Arizona State University.PortBench: a CAISI-developed non-public evaluation that assesses the ability of AI models to port command line interface (CLI) tools to different programming languages, given a reference implementation in one language. CAISI plans to release an in-depth description of PortBench in the future.FrontierScience<\/a>: a benchmark that evaluates expert-level scientific reasoning through international science olympiad problems and PhD-level, open-ended problems representative of sub-tasks in scientific research in physics, chemistry, and biology, developed by OpenAI.For descriptions of the other benchmarks, see Section 3 of CAISI\u2019s earlier Evaluation of DeepSeek AI Models<\/a>.<\/p>\n \u00a0<\/p>\n Capability Lag Measurement<\/p>\n CAISI uses an approach inspired by Item Response Theory (IRT) to determine the capability level of each evaluated model aggregated across the evaluated benchmarks. IRT was originally developed for psychometric testing, such as a setting where a group of students complete a number of exam questions, and the results are used to determine the relative competency of each student and the difficulty of each exam question. CAISI applies a similar technique for aggregate capability measurement by treating AI models as students and individual benchmark tasks as exam questions. For a fuller explanation, see the Appendix. As shown in Figure 1, the U.S. capability frontier tends to lead the PRC frontier by roughly 8 months.<\/p>\n \u00a0<\/p>\n Model Serving and Inference<\/p>\n CAISI served DeepSeek V4 from cloud-based H200 and B200 GPUs, and used developer-recommended settings for context length, max_tokens, temperature, top_p, preserving internal reasoning, system prompt, and maximum thinking. To rule out the presence of inference or configuration errors, CAISI reproduced the developer\u2019s self-reported benchmark results on GPQA-Diamond.<\/p>\n Agentic evaluations were conducted with Inspect\u2019s built-in ReAct agent<\/a>. Budgets were set to 1M weighted tokens for PortBench and CTF-Archive-Diamond, and 500k weighted tokens for SWE-Bench Verified. For the definition of weighted tokens, see Appendix A2 of CAISI\u2019s earlier Evaluation of DeepSeek AI Models<\/a>.<\/p>\n \u00a0<\/p>\n Comparison of DeepSeek and CAISI Evaluations <\/p>\n Figure 3: Comparison of DeepSeek V4 with frontier US models across two benchmark suites. DeepSeek’s technical report indicates DeepSeek V4 is competitive with frontier U.S. models across a range of benchmarks (Figure 3a). However, CAISI\u2019s evaluation of these models on benchmarks not featured in DeepSeek’s report show worse performance on some reasoning and agent-based evaluations, like the ARC-AGI-2 semi-private dataset, the held-out software engineering evaluation PortBench, and the cyber benchmark CTF-Archive-Diamond (Figure 3b).<\/p>\n \u00a0<\/p>\n DeepSeek V4 Costs Less Than Other Models Of Similar Capability<\/p>\n For the purpose of cost comparison, CAISI selected a U.S. reference model by filtering out U.S. models that performed significantly worse on public benchmarks or that cost significantly more per token than DeepSeek V4 Pro. The only model meeting these criteria was GPT-5.4 mini, which was selected as a point of reference. In CAISI\u2019s aggregated capability analysis, GPT-5.4 mini receives an Elo score of 749, which is similar to DeepSeek V4 Pro\u2019s score of 800.<\/p>\n DeepSeek V4 costs less than GPT-5.4 mini on 5 out of 7 CAISI benchmarks. On those 7 benchmarks, DeepSeek V4 ranged from 53% less expensive to 41% more expensive. Two CAISI benchmarks were excluded from cost comparisons: PortBench because it has continuous scoring which isn\u2019t yet supported by CAISI\u2019s cost comparison methodology, and ARC-AGI-2 because of technical issues with the GPT-5.4 mini evaluation run.<\/p>\n Figure 4: End-to-end expense of GPT-5.4 mini and DeepSeek V4 Pro on different benchmarks, for benchmark tasks that both models solve correctly.\u00a0 The following developer-reported token prices were used:<\/p>\n \u00a0<\/p>\n Input token cost (uncached)<\/p>\n Input token cost (with cache)<\/p>\n Output token cost<\/p>\n DeepSeek V4 Pro<\/p>\n $1.74 \/ 1M tokens<\/p>\n $0.0145 \/ 1M tokens<\/p>\n $3.48 \/ 1M tokens<\/p>\n GPT-5.4 mini<\/p>\n $0.75 \/ 1M tokens<\/p>\n $0.075 \/ 1M tokens<\/p>\n $4.50 \/ 1M tokens<\/p>\n Appendix<\/p>\n![\"Comparison](\"https:\/\/www.newsbeep.com\/us\/wp-content\/uploads\/2026\/05\/combined_benchmark_plot.png\")
<\/p>\n
(a) Benchmarks selected and reported by DeepSeek, where V4 appears roughly on par with frontier US models. (b) Benchmarks from CAISI\u2019s suite on which DeepSeek V4 lags behind U.S. models. CAISI pre-committed to its overall benchmark suite, i.e. did not select benchmarks on the basis of results.<\/p>\n![\"End-to-end](\"https:\/\/www.newsbeep.com\/us\/wp-content\/uploads\/2026\/05\/4-Cost-Comparison.png\")
<\/p>\n
Taller bars represent higher end-to-end cost. Values above 1.0 indicate DeepSeek V4 costs more than GPT-5.4 mini. Numbers within the bars denote the average expense a model incurred to solve a benchmark task\/question (where the average is computed over the set of benchmark tasks\/questions that both models solve correctly).<\/p>\n