brokenisfixed @brokenisfixed

Joined May 2008

Tweets

1K
Followers

28
Following

365
Likes

4K

Alex Prompter @alex_prompter

6 months ago

🚨 Holy shit… Stanford just published the most uncomfortable paper on LLM reasoning I’ve read in a long time. This isn’t a flashy new model or a leaderboard win. It’s a systematic teardown of how and why large language models keep failing at reasoning even when benchmarks say they’re doing great. The paper does one very smart thing upfront: it introduces a clean taxonomy instead of more anecdotes. The authors split reasoning into non-embodied and embodied. Non-embodied reasoning is what most benchmarks test and it’s further divided into informal reasoning (intuition, social judgment, commonsense heuristics) and formal reasoning (logic, math, code, symbolic manipulation). Embodied reasoning is where models must reason about the physical world, space, causality, and action under real constraints. Across all three, the same failure patterns keep showing up. > First are fundamental failures baked into current architectures. Models generate answers that look coherent but collapse under light logical pressure. They shortcut, pattern-match, or hallucinate steps instead of executing a consistent reasoning process. > Second are application-specific failures. A model that looks strong on math benchmarks can quietly fall apart in scientific reasoning, planning, or multi-step decision making. Performance does not transfer nearly as well as leaderboards imply. > Third are robustness failures. Tiny changes in wording, ordering, or context can flip an answer entirely. The reasoning wasn’t stable to begin with; it just happened to work for that phrasing. One of the most disturbing findings is how often models produce unfaithful reasoning. They give the correct final answer while providing explanations that are logically wrong, incomplete, or fabricated. This is worse than being wrong, because it trains users to trust explanations that don’t correspond to the actual decision process. Embodied reasoning is where things really fall apart. LLMs systematically fail at physical commonsense, spatial reasoning, and basic physics because they have no grounded experience. Even in text-only settings, as soon as a task implicitly depends on real-world dynamics, failures become predictable and repeatable. The authors don’t just criticize. They outline mitigation paths: inference-time scaling, analogical memory, external verification, and evaluations that deliberately inject known failure cases instead of optimizing for leaderboard performance. But they’re very clear that none of these are silver bullets yet. The takeaway isn’t that LLMs can’t reason. It’s more uncomfortable than that. LLMs reason just enough to sound convincing, but not enough to be reliable. And unless we start measuring how models fail not just how often they succeed we’ll keep deploying systems that pass benchmarks, fail silently in production, and explain themselves with total confidence while doing the wrong thing. That’s the real warning shot in this paper. Paper: Large Language Model Reasoning Failures

brokenisfixed @brokenisfixed

Alex Prompter @alex_prompter

vas @vasuman

GOG.COM @GOGcom

ICE T @FINALLEVEL

Respectful Memes @RespectfulMemes

poorly drawn lines @PDLComics

Respectful Memes @RespectfulMemes

Lub @Lub_De00

Jane Riekemann #FBPE @jriekemann

XKCD Comic @xkcdComic

brokenisfixed @brokenisfixed

Elon Musk @elonmusk

Quake @Quake

Respectful Memes @RespectfulMemes

brokenisfixed @brokenisfixed

brokenisfixed @brokenisfixed

brokenisfixed @brokenisfixed

brokenisfixed @brokenisfixed

Respectful Memes @RespectfulMemes

Ermeasee @Ermeasee442483

Biocabel @Black_hat_tech

Ole 🇨🇦 🇱🇹... @OleQuestionable

Oleg Rumiancev @olezhka_lt

Yona @yonarbi

Aoomle @aoomle

Jayadevan TR @Jayadevan_TR

🦇bat attack🦇 @salixpurple

Simon Kuestenmacher @simongerman600

Code Motivation @code_motivation

Donatas Kulvičius @donkulv

| AEROKINETIC | ¯\_(... @AEROKINETIC11

Gytis Greitai @gytisgreitai

Mindaugas Pelanis @mpelanis

Ana Shackley @ashleighwp9

Marius Kazlauskas @UgiBugiVugi

BO3 Luck @aquiregame

kasu 🇱🇹💛💙 @kasuparu

Linux Legion @Linux_Legion

Youtube Showcase @TwichFollows

Eimantas S. @eimants

Zeina Scarlet Sabra @Zeinascarlet

nicolai @MurphyNicolai

Dr Maeve O'Connell RM... @thelovelymaeve

VitaMasaliene @vitamasaliene

Rosita Amber @RositaAmber

Freiheitsgrade.com @Freiheitsgrade_

S C @starcraft2music

David Jaffe @davidscottjaffe

Grummz @Grummz

TheStandupPod @thestanduppod

Geoff Keighley @geoffkeighley

Carl Jung Archive @QuoteJung

emi ⭐️ @emiru

Daniel Hooper @DanielcHooper

IT Unprofessional @it_unprofession

Valentin Ignatev @valigo

Raphael Luba @LubaRaphael

Fireship @fireship_dev

Scott Adams @ScottAdamsSays

ThePrimeagen @ThePrimeagen

Ryan Fleury @rfleury

Proton Mail @ProtonMail

Łukasz | Wookash Pod... @wookash_podcast

Casey Muratori @cmuratori

Low Level @LowLevelTweets

Vjekoslav Krajačić @vkrajacic

Secretary Kennedy @SecKennedy

Тsфdiиg @tsoding

terminally onλine ε... @tekbog

The DarkHorse Podcast @thedarkhorsepod

Gad Saad @GadSaad

Robert F. Kennedy Jr @RobertKennedyJr

Konstantin Kisin @KonstantinKisin

Zack @Asmongold

Imogen Heap @imogenheap

Bjorn Lomborg @BjornLomborg

World of Statistics @stats_feed

HOW THINGS WORK @HowThingsWork_

Dr. Debra Soh - SEXTI... @DrDebraSoh