ARC-AGI-2: Abstract Reasoning and Fluid Intelligence

The ARC-AGI-2 benchmark (Abstraction and Reasoning Corpus for Artificial General Intelligence, version 2) tests abstract pattern recognition and fluid intelligence — the ability to identify novel visual patterns and extrapolate rules from minimal examples. Unlike knowledge-heavy benchmarks, ARC-AGI-2 is specifically designed to resist memorization and training data contamination. [8]

Gemini 3.1 Pro achieves 77.1% on ARC-AGI-2, while Claude Opus 4.6 scores 68.8% — an 8.3 percentage point gap that represents the largest single benchmark advantage either model holds over the other. This result aligns with Gemini’s multimodal architecture, which processes visual patterns natively rather than through text-mediated reasoning. [8]

The significance of this gap extends beyond the benchmark itself. ARC-AGI-2 is widely considered the closest existing proxy for measuring progress toward general intelligence. Gemini’s lead suggests that Google’s multimodal-native architecture may have fundamental advantages in pattern recognition tasks that text-specialized architectures cannot easily replicate. [8]

GPQA Diamond: Expert-Level Academic Reasoning

GPQA Diamond (Graduate-Level Google-Proof Q&A, Diamond subset) tests expert-level reasoning across physics, chemistry, biology, and mathematics. Questions are crafted by domain experts to be resistant to simple search or memorization — they require multi-step reasoning chains and genuine understanding of underlying principles. [7]

Gemini 3.1 Pro scores 94.3% versus Claude’s 91.3% on GPQA Diamond. Both scores are remarkably high — well above human expert baselines — and the 3.0 percentage point gap is narrower than ARC-AGI-2. At these performance levels, both models demonstrate near-expert proficiency in graduate-level scientific reasoning. [7]

The practical implication is that for enterprise applications involving scientific analysis, technical problem-solving, or academic-level reasoning, both models are highly capable. The marginal Gemini advantage on GPQA Diamond is unlikely to be the deciding factor in platform selection for these workloads. [7][15]

The HLE Scoring Controversy

The Humanity’s Last Exam (HLE) was designed as the ultimate benchmark — a collection of questions so difficult that only the world’s top domain experts could answer them. It was intended to be the evaluation that no AI model could saturate, providing years of meaningful discrimination between increasingly capable systems. [10]

Then the scoring controversy detonated. Anthropic reported that Claude Opus 4.6 achieved 66.6% on HLE — a score that would represent an extraordinary leap in AI capability. Independent evaluators subsequently tested the same model on the same benchmark and found a score of approximately 18.6%. [10]

The 48 percentage point gap between vendor-reported and independently-measured scores is unprecedented in AI evaluation history. It immediately raised fundamental questions about benchmark methodology, evaluation integrity, and the reliability of vendor-reported performance claims. [10]

Several explanations have been proposed for the discrepancy:

Evaluation methodology differences: Anthropic may have used multiple attempts per question, selected best-of-N responses, or applied scoring criteria that differ from the benchmark’s standard methodology. If Anthropic’s evaluation allowed the model 10 attempts per question and counted any success, while independent evaluation used single-attempt scoring, the gap is mechanically explained. [10]

Test set selection: It is possible that Anthropic evaluated on a subset of HLE questions rather than the full benchmark, potentially one that was more favorable to its model’s capabilities. [10]

Dataset contamination: The most damaging hypothesis is that HLE questions or closely related content appeared in Claude’s training data, effectively turning a reasoning test into a recall test. This explanation, while not proven, is consistent with the pattern of vendor-reported scores dramatically exceeding independent evaluation. [10][12]

GDPval-AA: Where Claude Wins Decisively

Not all benchmarks favor Gemini. The GDPval-AA evaluation — reported by Artificial Analysis as a composite quality metric — gives Claude Opus 4.6 a score of 1,606 compared to Gemini’s 1,317. This 289-point gap (approximately 22%) represents Claude’s most decisive benchmark advantage. [7]

GDPval-AA measures overall output quality, coherence, and accuracy across a diverse set of real-world tasks. Unlike specialized benchmarks that test narrow capabilities, it attempts to capture the holistic user experience of interacting with the model. Claude’s significant lead here aligns with its philosophy of depth-first intelligence — producing higher-quality individual outputs rather than optimizing for breadth of capability. [7]

GPT-5.3-Codex: The Coding Benchmark Disruptor

OpenAI’s GPT-5.3-Codex introduces a third competitor that reshapes the benchmark landscape, particularly in coding and cybersecurity evaluations. Released in February 2026, it dominates several benchmarks that Claude and Gemini were previously contesting between themselves. [27]

On Terminal-Bench 2.0, GPT-5.3-Codex scores 77.3% — surpassing both Gemini’s 68.5% and Claude’s 65.4% by wide margins. This benchmark tests autonomous terminal operation, and Codex’s cloud sandbox architecture (where it operates with full filesystem and shell access in isolated containers) gives it a structural advantage. [27]

On the new CyberSec CTF benchmark, GPT-5.3-Codex achieves 77.6%, earning it the first “High” cybersecurity classification under OpenAI’s Preparedness Framework. This score reflects the model’s ability to autonomously identify vulnerabilities, exploit web applications, and solve capture-the-flag security challenges — capabilities that are as much a safety concern as a benchmark achievement. [27]

OSWorld (computer-use autonomy) at 64.7% and SWE-Bench Pro (harder variant of SWE-bench) at 56.8% complete the picture. GPT-5.3-Codex is not the highest-scoring model on all benchmarks, but it establishes a new category of evaluation: autonomous agentic task completion measured over minutes to hours, not seconds. [27]