Arena.ai

Claude Code vs Codex vs Pi: which coding agent wins? @melissapan, PhD candidate at UC Berkeley’s Sky Computing Lab and previous @arena intern, explored the hidden “harness tax”: how the system surrounding an AI model affects its cost and performance. She reports three surprising findings. One: harness choice impacts cost more than accuracy. To learn about all three findings, and what they mean for building useful coding agents on realistic budgets, check out the full video here: https://youtu.be/4R8qZGtEnAE

Real-world results are in for Claude Sonnet 5.5 (High) by @AnthropicAI. It just landed #4 in Code Arena: WebDev with 1699 pts, and has reshaped the Pareto frontier with its cost efficiency! Claude Sonnet 5.5 (High) delivers nearly top performance at a blended $8 per Mtoken, reshaping the Pareto frontier! This model is 80% cheaper than both Claude Fable 5.1 (Max) in the #3 spot overall, and GPT-6 Astra (Max) at #2. See Pareto placement below. Overall, Claude Sonnet 5.5 (High) is a +159 pt improvement from Sonnet 5 (High) at #37 with 1540 pts. This gain compared to its previous variant also shows up across these key domains so far: - Reference-Based Design: #38 → #4 - Simulations: #37 → #4 - Gaming: #36 → #4 Congrats to @AnthropicAI on this release!

https://x.com/i/article/2104955648664584192

Update to the Agent Arena Pareto frontier: GPT-6 Luna (Max) has landed on the Agent Arena Pareto frontier with +1.59% net improvement at a median cost of just $0.05 per task! Luna costs 29% less than DeepSeek-V4.1-Flash and 76% less than Hy4 preview, while trailing their net-improvement scores by just 2.27 and 2.51 percentage points respectively. Compared to other GPT models on the Pareto frontier, GPT-6 Luna (Max) costs 94% less than GPT-6 Sol (Max), and 98% less than GPT-6 Astra (Max). Congrats to the @OpenAI team on this cost-efficient release!

Ask an AI model to pick the better answer, and it'll usually pick its own. We compared 34,580 verdicts from 12 models with human votes across 1,460 battles on Arena. The results show that AI judges have their own taste, and it's unlike ours. They: - Favor their own answers. On average, a model picked its own answer 58% of the time. People picked that same answer 34% of the time. GPT-6 Astra picked itself 88% of the time. - Rarely call a draw. People called a tie or "both bad" in 32% of battles. GPT-5.6 Sol picked a winner 96% of the time. - Side with each other over people. They agreed with other AIs 79% of the time and with people 57% of the time. Every judge did, by a margin of 18 to 27 percentage points. Full results in the article from @DawidGalarowicz below.

GPT-6.1 by @OpenAI is now in the Arena! Head to Agent Arena to test it out, and your votes will shape its evaluation. Scores coming soon. In Agent Arena, we measure models on millions of real-world, long-horizon agentic tasks. Models can access web search, filesystem, and terminal tools to complete complex workflows. The leaderboard measures model performance on outcomes relative to the average model using a causal tracing methodology. GPT-6.1 is also available in Code Arena: WebDev, Text, Vision, Search and Document.

For a limited time starting September 30 at 8am PT, Claude Sonnet 5.5 by @Anthropic will be available to test directly on Arena! Once access is live, head to Direct Mode and select it from the dropdown menu. Stay tuned tomorrow for the direct link. Claude Sonnet 5.5 (High) will be available in Direct Mode for 48 hours. It will remain available in Battle and Agent Mode after that, as it already is today.

Watch the progress of frontier models in bringing Ancient Rome to life on Arena, from Q4 2025 to now. Scores for Claude Sonnet 5.5 by @AnthropicAI and GPT-6.1 by @OpenAI are coming soon. Real-world tasks from our global community of users power the Arena leaderboards. Head to Arena now to test it out, and stay tuned! Featured models: - Claude Opus 4.5, 4.6, 4.7, 4.8, 5 and 5.5 - Claude Fable 5 and 5.1 - Claude Sonnet 5.5 - GPT-5.2, 5.3 Codex, 5.4, 5.5, 5.6 Sol, 6 Astra, and 6 Sol - Gemini 3.1 - Kimi K3 - Qwen 3.8 Max Find the prompt from @petergostev below.

Introducing the Arena Alignment Index, our new benchmark measuring safety and alignment of AI agents in real-world use. Built from 90K+ real-world agent sessions across 27 models, the index measures three critical signals: - Unauthorized Action (UA): Taking actions beyond the user's instructions or permissions - False Attribution (FA): Attributing statements or actions that are contradicted by user-provided evidence - Deceptive Completion (DC): Claiming a task was completed when it was not. Key findings: - OpenAI models currently lead the Alignment Index - Rogue actions are rare, but can have serious consequences when they occur - Agents can mislead users about task progress - Misalignment risks increase with conversation length - Safety and alignment are improving across model generations As shown in the leaderboard below (sorted by lab), @OpenAI’s GPT-6.1-Sol leads the Arena Alignment Index with a score of 87.9, followed by @AnthropicAI’s Claude-Opus-5.5 at 83.2 and @SpaceXAI's Grok-4.7 at 82.7. OpenAI also has the best observed rates across all three signals: 0.89% Unauthorized Action, 1.98% False Attribution, and 2.34% Deceptive Completion. Across all four labs, newer models consistently outperform their predecessors, suggesting broad progress in agent safety and alignment. As agents take on longer, more complex, and higher-stakes tasks, measuring not just what they can accomplish, but how safely and reliably they act, becomes increasingly important. This marks an important step toward making safety and alignment a core part of how Arena evaluates AI. The index is an initial starting point, and we'll continue expanding the index with additional safety signals and models over time. More analysis below👇

Today, we're announcing our Series B: $200M at a $3.1B valuation, and the release of Arena's Alignment Index. AI is advancing faster than our ability to evaluate it. The world needs a neutral third party to measure how safe and aligned AI actually is once it's in the hands of real people. That's the role Arena is stepping into today. The Alignment Index is built from real-world agent traces, starting with three signals: Unauthorized Action, False Attribution, and Deceptive Completion, with results for over 20 frontier models launching today. Hear more from our CEO @ml_angelopoulos below, and read the full breakdown, and more details about our company growth in thread. This funding will allow us to go further on safety signals, agentic capabilities, and modalities. It will also allow us to grow the company to match the scale of our mission. Today, we’re still a nimble team of 90, and we're looking for people to join us across research, product, engineering, and more. Come build with us. As AI gets more capable and autonomous, understanding how well it can safely and effectively work for people only matters more. That's what Arena is built to measure and advance. Thank you to our community, our investors, and everyone building with us! Take a look at our open roles if you want to join us (link in bio).

Exciting news: Gemini 4 Carbon (High) from @GoogleDeepMind just made a huge debut on Arena, landing #1 in Text Arena, #2 in Agent Arena, and #4 in Code Arena: WebDev. Back at the top, Gemini 4 Carbon delivers frontier intelligence and reshapes the Pareto frontier across three arenas. In Agent Arena, Gemini 4 Carbon ranks #2 with a +16.2% net improvement across +21.9K real-world agentic sessions from our global community, surpassing top models GPT-6 Astra and Fable-5.1. At $1.80 median cost per task, Gemini 4 Carbon (High) costs 72% less than the #1 ranked Claude Opus 5.5 (Max). Its standouts by key signals: - #1 Steerability (+18.6%): can the model course-correct when you push back? - #2 Praise vs Complaint (+37.9%): implicit sentiment in users' reactions - #3 Confirmed Success (+18.7%): explicit “yes, that worked” from users In Code Arena: WebDev, Carbon scored 1768 pts landing #4: - Just 2 pts below #3 Claude Sonnet 5.5 (xHigh), at 1770 pts - 89 pts above #11 Gemini 4 Argon (High), at 1679 pts By category, it ranks #3 in Gaming and Simulations, and #5 in Brand & Marketing. In Text Arena, Gemini 4 Carbon (High) is the new #1 with 1528 pts, now ahead of Gemini 4 Argon (High) at #2 with 1520 pts. Congrats to the @GoogleDeepMind team on this huge milestone!

Agent Arena update: we’ve evolved the Bash Recovery signal, with Bash Reliability. Here’s how it impacts the leaderboard. At the top for Agent Arena: - #1 Claude Opus 5.5 (Max), @AnthropicAI holds the top - #2 GPT-6.1 Sol (Max) by @OpenAI - #3 Claude Fable 5.1 (Max) by @AnthropicAI - #4 GPT-6 Astra (Max) by @OpenAI - #5 Gemini 4 Argon (High) by @GoogleDeepMind The first 14 spots are all proprietary models with the best open weights models being GLM 5.2 at #15, followed by MiMo V2.6 Pro at #18. Read more about Bash Reliability and check out the Agent Arena leaderboard in thread.