Che-Ping Tsai @chepingt

Introducing factuality in the Arena: a new ranking of models according to a weighted combination of human preference and factuality. Model rankings are now viewable according to a weighted combination of human preference and factuality. Factuality is live in our Text and Search Arenas as a non-default toggle. We audit model responses by randomly sampling battles and extracting web-verifiable claims. We then verify these claims and compare the average correctness between model responses. To power these rankings, we’ve labeled over 2 million claims made by LLMs in real-world conversations, 1.3+ million from Text Arena, and 700k+ from Search Arena. Notable highlights with factuality enabled in the Text Arena: - Claude Fable 5 moves down slightly to spot #2 - GPT-5.5 saw the largest increase, moving up 13 spots into the #7 spot - Muse Spark dropped the most from #7 to #20 (-13pt) By labs, Meta saw the largest drop from #2 to #5, while Anthropic overall held the #1 spot. Looking at only open model providers, Xiaomi saw the largest improvement, jumping from #9 to #6. Learn more about the findings and methodology in this thread.

Some big updates to GameDevBench! We’ve tripled (!) the benchmark to 333 tasks and updated the paper accordingly. We also added GLM-5.2 and Opus 4.8 to the leaderboard. Opus now leads by a small margin, while GLM-5.2 is the strongest open-source model. This is despite not having any visual capabilities! Agentic game development remains far from solved, but it’s also exciting to see more agentic game dev work starting to emerge. I’m hoping we’ll see rapid progress here soon.

@valeriechen_ @sancortes_95 Congrats! Happy for you!

As my time in Pittsburgh comes to an end... I'm excited to share that I will be joining UIUC as an Assistant Professor in fall 2027! I will recruit PhD students in both ECE and CS. I’m looking for students and postdocs who are excited to continue building towards collaborative AI systems that learn and adapt through interaction.

@aryanvichare10 @arena Let’s gooooo

Belated career update: I graduated from CMU MLD and have been working at @Arena on AI evaluation. Fair and principled measurement has long been at the heart of my PhD research, and I’m excited to continue pursuing this direction in the context of agentic AI. Grateful to be part of the team at Arena.

Arena.ai @arena

2 months ago

Introducing Agent Arena: real-world agentic evals at scale. How do you evaluate agents doing actual work? We measure millions of live sessions where real users accomplish real tasks. On Arena, models now get web search, filesystem, and terminal tools to complete complex

75 148 1K 601K 325

Today we’re excited to launch Agent Arena - a major step towards evaluating AI in the agentic era. The frontier is no longer just about chatbots answering questions. It is about completing real tasks: using tools, adapting to user feedback, navigating errors, and producing useful artifacts. At @Arena, we believe the best evaluations should be grounded in real-world use: production workloads, live user interactions, and the actual utility created through real work. Agent Arena puts models in live tool-use environments, captures rich signals from user-agent interaction, and uses causal inference to measure performance across millions of real agentic traces - orders of magnitude larger than static benchmarks. It is both a live leaderboard that continuously evolves with frontier AI uses, and a cutting-edge agentic AI product for everyone. Huge shoutout to the @Arena team for the incredible work. This is the biggest project at Arena ever, a massive cross-functional effort across research, engineering, and product to mark a major transformation of the platform. If you’re excited about this mission, come build with us!

Arena.ai @arena

2 months ago

Introducing Agent Arena: real-world agentic evals at scale. How do you evaluate agents doing actual work? We measure millions of live sessions where real users accomplish real tasks. On Arena, models now get web search, filesystem, and terminal tools to complete complex

75 148 1K 601K 325

@iamwaynechi @arena LFG🐎

@ml_angelopoulos @arena 🚀🚀🚀

@iamwaynechi @arena @ml_angelopoulos @infwinston @istoica05 Welcome to Arena🚀🚀🚀

Introducing 7 new leaderboard views for frontend output in Code Arena. Aggregate leaderboards don’t tell the full story. "Best frontend coding model" depends on what you're building, so we built leaderboards that show exactly that. After analyzing 250,000+ Code Arena prompts, we identified the major frontend web development task categories: - Brand & Marketing - Reference-Based Design - Data & Analytics - Consumer Product - Gaming - Simulations - Content Creation Tools With this release, @AnthropicAI is a big winner as it has at least 1 model in top 4 spots across all 7 categories. But there’s more to the story in the margins. Dig into the thread to see exactly which models are currently on top of each domain.

Models are typically specialized to new domains by finetuning on small, high-quality datasets. We find that repeating the same dataset 10–50× starting from pretraining leads to substantially better downstream performance, in some cases outperforming larger models. 🧵

A year of cooking 👨‍🍳and we’re finally serving Mamba-3. What began as a small effort to revisit a few recurring limitations of SSMs grew into a much bigger project. Taking a more principled state space perspective ended up tying these threads together.

Albert Gu @_albertgu

5 months ago

The newest model in the Mamba series is finally here 🐍 Hybrid models have become increasingly popular, raising the importance of designing the next generation of linear models. We've introduced several SSM-centric ideas to significantly increase Mamba-2's modeling capabilities

42 309 2K 451K 834

@dylanjsam @yus167 @OpenAI @zicokolter @andrew_ilyas @furongh Congrats!

I'll admit, going in I was not 100% sure this was possible: we trained a tiny 4B model (QED-Nano) to prove math theorems at the Olympiad level! Today, we release the full recipe, from the data curation done for SFT to our RL algorithm that explicitly optimizes for test-time scaling over millions of tokens (i.e., we train QED-Nano to continually improve as we apply modern day test-time scaffolds like DeepSeekMath-agent over it). 🧵⬇️

RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) 🧵

We run online RL on a mixture of problems: some are easy to explore (high pass rate), and some are very very hard (need to sample A LOT before we see any positive sample). Turns out RL on such a mixture can lead to a "rich-gets-richer" effect, where RL over-sharpens on the easy problems, at the cost of getting stuck in a "plateau" on harder ones, making it even harder to sample a correct trace on those. RL literature calls this "ray interference". In our recent work POPE, we show that using privileged info. to guide exploration on hard problems can tackle ray interference! 🧵⬇️

RL training of LLMs spends tons of compute on sampling rollouts 🤖💸 But most runs are YOLO 🤟, telling us little about how to scale sampling compute optimally. Given a fixed sampling compute budget, how should we allocate it across: • sequential iterations ⏩ • parallel rollouts 🎲 Answers to this with scaling laws 📈 and more in our new blog post ⬇️

1/⚠️ Parallel test-time scaling (e.g., pass@k) usually wastes compute - models often repeat the same dominant failure❌ How should we effectively generate creative solutions? While typical methods such as increasing temperature 🌡️ usually fail, we put forward Mode‑Conditioning (ModC) - a simple yet powerful training and test-time framework that allocates compute across diverse reasoning modes🎨We show that ModC largely improves pass@k across SFT, distillation, and RL settings. With ModC, we get 4-8x efficiency gains in math reasoning using the same training data!

Worked closely with Burak on SSL—super thoughtful, rigorous, responsible and supportive. Strongly recommend them for any academic position!

Burak Varıcı @VariciBurak

9 months ago

📢I’m on the academic job market!📢 I mainly work on representation learning and causality (CRL, identifiable SSL, causal discovery, robotics applications, rep. learning for tabular data) Also, I’ll be at #NeurIPS Dec. 2–7, reach out to chat about any of the above :)

0 3 19 2K 1