Yang Fan Yun @YangFanYun

Today we're releasing Composite-Bench: a long-horizon computer-use benchmark with certified-optimal answers and verified compute. The strongest open-weights model isn't Kimi K3. GLM-5.2 beats it by 32 points and clears every closed model we tested except Claude.

93 53 604 96K 199

View Details

Yang Fan Yun @YangFanYun

2 weeks ago

@rshanreddy WOO likewise shan!!

0 0 0 75 0

View Details

Yang Fan Yun @YangFanYun

3 weeks ago

@andyzhang WOO thank you andy!!

0 0 0 156 0

View Details

Yang Fan Yun @YangFanYun

3 weeks ago

@siddsax WOO thank you @siddsax !

0 0 1 268 0

View Details

Yang Fan Yun @YangFanYun

3 weeks ago

@dotgil most from the former!

1 0 1 844 0

View Details

Yang Fan Yun @YangFanYun

3 weeks ago

Thanks Kamil! It's in Appendix C of the blog post but the TLDR is: we're measuring exact optima rather than mean rewards, which disproportionately affects Sol's tendency to produce strong answers that stop short of the exact optimum. Same thing affects Grok. However, even on the axes that ignore exact optima, Fable stays ahead of Sol (e.g., on NPP), so the gap reflects a genuine edge in deliberate long-horizon planning.

0 0 2 450 0

View Details

Yang Fan Yun @YangFanYun

3 weeks ago

We've explained this in the blog post - the TLDR is that we're measuring exact optima rather than mean rewards, which disproportionately affects Sol's tendency to produce strong answers that stop short of the exact optimum. Same thing affects Grok. However, even on the axes that ignore exact optima, Fable stays ahead of Sol (e.g., on NPP), so the gap reflects a genuine edge in deliberate long-horizon planning.

0 0 0 141 0

View Details

Yang Fan Yun @YangFanYun

3 weeks ago

@garzavilo TLDR Grok's final answers are satisficing - they were consistently pretty good (0.73–0.92 median per slice) but but rarely optimal, leading to a low macro optimal success rate.

1 0 0 634 0

View Details

Yang Fan Yun @YangFanYun

3 weeks ago

Grok was really interesting - it's final answers are consistently good but rarely optimal. Mean reward was high, but the success rate was low. Fable never refused once in our eval because we've intentionally made these safe for public release (no PII, auth, privacy/security issues). We talk about it more in the blog post!