Nicolas Zucchet @NicolasZucchet
Postdoc @Stanford, prev. PhD @CSatETH | student researcher @GoogleDeepMind | @Polytechnique nicolaszucchet.github.io Stanford, CA Joined December 2017-
Tweets239
-
Followers773
-
Following453
-
Likes1K
What happens when we remove the nonlinear activation from a (very) sparse MoE layer? Surprisingly, perplexity per FLOP improves and we get access to remarkably interpretable structure in the transformer FFN layers without having to train any SAEs/Transcoders/... 🧵👇
@vatsalbaherwani Very cool experiments and results! We do have a theory for why context length and sparsity speed up learning in arxiv.org/abs/2505.17863 (Eq 6). And we found similar effects of attention patching (using trained ones instead of GT) in arxiv.org/abs/2503.21676 (S2.2)!
I'm excited to share what we're building at Engram! This team is incredible, and we're working on one of the most interesting problems in AI right now: how to build models that are tailored to each person and continually learn from experience. Come join us!
Announcing our COLM 2026 workshop: Scientific Understanding of Foundation Models: we invite submissions on training dynamics, scaling laws, data and optimization, post-training, reward modeling, evaluation science, reliability, reproducibility, and theoretical understanding of foundation models. We especially welcome rigorous empirical studies, theory-grounded work, negative results, reproductions, and papers that bridge theory and practice for contributing to this goal. 📍 In person at COLM 2026, San Francisco 🗓️ Submission deadline: June 23, 2026, 11:59 PM AoE 🌐 lnkd.in/eEU5EVt4 🧑🏫 Speakers: @SuryaGanguli, @JikaiJin2002, @zhiyuanli_, @waterluffy, @valentina__py, @lschmidt3, @MohammadShoeybi, @andrewgwils.
I've had access to Fable for a bit. A genuine jump in capability, I could feed it a 15 page design document for a project and it would work for 9+ hours and deliver terrific results. But working with it is weird & weirder is coming Lots of examples: open.substack.com/pub/oneusefult…
@andrewgwils Depends imo, totally fine to use reductions when some behaviors have been well identified empirically to better understand them, but things get pretty different when trying to come up with a more general theory.
What are the real problems to be solved in continual learning? In my latest post, I tackle this question — reviewing where I think the field went astray in the past, how language models changed things, and where the real challenges remain. 1/2
@francoisfleuret I've been dreaming about this for years, and unfortunately made very little progress. Looking forward to what comes out!
@mrtnm @andrewgwils @dongkyucho @ShikaiQiu @rumichunara @Pavel_Izmailov @ZeyuanAllenZhu @karpathy One hypothesis would be that replay works to maintain performance on everything that is not pure knowledge. If you end up investigating that, I'm curious to know the results!
@lateinteraction thanks for the clarification! one way to formalize your argument could be with macro (e.g. proof strategy) vs. micro (e.g. calculation) mistakes. OPD indeed unlikely to fix macro mistakes, but efficient at correcting micro mistakes than RL because of direct credit assignment
Why does deep learning generalize? What does weight decay really do? Can algorithmic information theory address these questions? In my latest preprint, I give a proof that the minimum neural weight norm matches the minimum program length (aka Kolmogorov Complexity), up to a logarithmic factor. In other words, the neural network with the smallest possible weight norm (that fits the data) must encode the shortest program (that fits the data). The result only holds for fixed-precision neural nets: infinite precision nets can store infinite information with finite (small) weights. arxiv.org/abs/2605.10878
🚨 Paper Alert: “RL Finetunes Small Subnetworks in Large Language Models” From DeepSeek V3 Base to DeepSeek R1 Zero, a whopping 86% of parameters were NOT updated during RL training 😮😮 And this isn’t a one-off. The pattern holds across RL algorithms and models. 🧵A Deep Dive
@SophieLWang Interesting finding! Any idea where this is coming from? e.g. difference from on- vs. off-policy, an artifact of post-training
@ziv_ravid The constant depends on the architecture to some extent, see Physics of LLMs 3.3 for discussion on that point arxiv.org/abs/2404.05405
We found a new way to get language models to reason. 🤯 No RL, no training, no verifiers, no prompting. ❌ With better sampling, base models can achieve single-shot reasoning on par with (or better than!) GRPO while avoiding its characteristic loss in generation diversity.
@BingbinL @KimberlyLiu0105 @scifordl @ezra_edelman @SurbhiGoel_ Very cool! Great to see some work in that direction. You may find some of our (and other) work interesting: arxiv.org/abs/2503.21676 arxiv.org/abs/2505.17863 arxiv.org/abs/2602.11149
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics! We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics. 🔨 arxiv.org/pdf/2604.21691 🔧
happiness @happinesslkai
2 Followers 224 Following
Andrei Nicolicioiu @anNicolicioiu
574 Followers 763 Following PhD student @Mila_Quebec, visiting researcher @AIatMeta studying LLM reasoning, robustness, causal ML. Past @MPI_IS @Bitdefender.
Michael Y. Li @michaelyli_
802 Followers 400 Following CS PhD @StanfordAILab @StanfordNLP advised by @noahdgoodman and Emily Fox. Prev: undergrad @princeton
carlolope_534 @carlolope_534
805 Followers 6K Following
Souradip Chakraborty @SOURADIPCHAKR18
2K Followers 5K Following Postdoc at @MIT CSAIL @nlp_mit Research: #RL, #Posttraining, #Alignment #RLVR Past: CS PhD @umdcs @ml_umd SR @GoogleResearch #JPMC #Walmart
Blake Bordelon ☕️... @blake__bordelon
2K Followers 891 Following Postdoc @Harvard CMSA, starting UT Austin FL 2026. 2:48 marathon
Adam Marblestone @AdamMarblestone
11K Followers 7K Following Technologist, Scientist Co-founder and CEO @Convergent_FROs Tweets do not represent employers
Bingbin Liu @BingbinL
1K Followers 339 Following Research Fellow at the Kempner Institute at Harvard University.
Aleksandra Bakalova @... @abakalova13175
128 Followers 173 Following (also go by Sasha) PhD student https://t.co/cO2r86TIPj
ollivier @ollivier26
40 Followers 784 Following
wormhole @wapurun
18 Followers 976 Following stop living in Minecraft survival mode, start living in creative mode
Ufk Asln 2773 @AslnUfkk
30 Followers 145 Following
Josh Lunger @joshjlunger
14 Followers 81 Following Computational neuroscience researcher @ Stanford, incoming PhD student in computer science. Building something new at South Park Commons.
Shichang (Ray) Zhang @ShichangZhang
503 Followers 303 Following Postdoc @Harvard | Research on XAI | Ex-@UCLA, Ex-@Stanford, Ex-@Berkeley
Huy Nguyen @huynt07
33 Followers 895 Following Interested in building online agents that adapt, evolve, and discover in open-ended worlds.
Ye Zhang @yezhang1998
16K Followers 571 Following Doing research about crypto, system and hardware | 中文碎碎念 @yezhang_cn
Mayalen Etcheverry @mayalen_etc
491 Followers 398 Following Exploring self-organizing systems and new AI paradigms @GoogleResearch
Ajitesh Shukla @ajitesh_shukla7
2K Followers 7K Following Student,Love to solve hardest math problem.LLM's,Mathematical Research(Geometric Topology,Differential Geometry,Algebraic Geometry).Lord Krishna is God Of Math
Chris Glaze @chris_m_glaze
2K Followers 4K Following Principal Research Scientist at @SnorkelAI. PhD in computational neuroscience. Previously: @penn @UofMaryland
timur kharisov @normisdeath
81 Followers 758 Following DL optimisation researcher at day, amateur artist at night
Jerry Liu @jerrywliu
491 Followers 661 Following ML & numerics @HazyResearch | ICME PhD at Stanford, @doecsgf fellow | prev @duolingo @berkeleylab @livermore_lab
Alice Turlen @AliveTurtleDove
2 Followers 401 Following
Pietro Astolfi @Piovrasca
347 Followers 818 Following
Akhil Vijaykumar @AkhilV83663705
43 Followers 5K Following
Soria Jules @jsor_ia
35 Followers 896 Following PhD Candidate at @UnivParisSaclay // @CEA_List Working on trustworthy & explainable AI. Opinions are my own.
Unknown @45676556787667h
41 Followers 767 Following
Lakshya A Agrawal @LakshyAAAgrawal
5K Followers 4K Following PhD @ UC Berkeley | @gepa_ai Creator | Created https://t.co/YxPZsXZJeS | AI4Code Research Fellow @MSFTResearch | Hobby Saxophonist
soda @private_soda
13 Followers 2K Following
Jinwoo Kim @jw9730
675 Followers 1K Following Postdoc at KAIST and previous visiting scholar at NYU, studying deep learning.
Doraking @doraking_en
283 Followers 7K Following May the Quantum be with you / AI Developer / Quantum Computing, AI, Blockchain / @Meeplia Founder / DM for collabs📩
Josh Ryan @jryandx
11 Followers 2K Following
Ronin V @RoninV_zen
22 Followers 92 Following
Francesco Cagnetta @Fraccagnetta
512 Followers 507 Following Theoretical physicist (bow tie included), inherently out of equilibrium. Studying data structure and deep learning. Marie Skłodowska-Curie fellow at @SISSA.
Maria Ios Glarou @_mar_g_
8 Followers 526 Following
Pinaki Roychoudhury @fsja_ajau_13145
0 Followers 4K Following
Ziming Liu @ZimingLiu11
14K Followers 886 Following Assistant Professor @Tsinghua_Uni CollegeAI. Chief Scientist @MetaCircle_AI Science of AI, AI for AI, AI for Science, KAN.
Kaitlin Maile @kaitlinmaile
439 Followers 585 Following Researcher @GoogleResearch Paradigms of Inteligence team
Goodfire @GoodfireAI
25K Followers 29 Following Using interpretability to understand, learn from, and design AI.
Andrei Nicolicioiu @anNicolicioiu
574 Followers 763 Following PhD student @Mila_Quebec, visiting researcher @AIatMeta studying LLM reasoning, robustness, causal ML. Past @MPI_IS @Bitdefender.
Jack Morris @jxmnop
53K Followers 1K Following research @engramlab // language models, information theory, science of AI // formerly @cornell
Michael Y. Li @michaelyli_
802 Followers 400 Following CS PhD @StanfordAILab @StanfordNLP advised by @noahdgoodman and Emily Fox. Prev: undergrad @princeton
Prophesee @Prophesee_ai
2K Followers 573 Following Prophesee reveals the invisible to machines with the world’s most advanced neuromorphic vision sensing and AI technologies.
Matthieu wyart @MatthieuWyart
972 Followers 109 Following Theoretical physics × machine learning Scaling laws | statistical mechanics | learning theory Prof @ Johns Hopkins & EPFL Trying to understand why LLMs work
Phillip Isola @phillip_isola
21K Followers 175 Following Associate Professor in EECS at MIT, trying to understand intelligence.
Sophie Wang ✈️ IC... @SophieLWang
1K Followers 140 Following Studying intelligent representations @MIT_CSAIL
Adam Marblestone @AdamMarblestone
11K Followers 7K Following Technologist, Scientist Co-founder and CEO @Convergent_FROs Tweets do not represent employers
Bingbin Liu @BingbinL
1K Followers 339 Following Research Fellow at the Kempner Institute at Harvard University.
Jamie Simon @learning_mech
1K Followers 75 Following doing fundamental science of deep learning | PhD from Berkeley | can catch a whole egg in my mouth
ClaudeDevs @ClaudeDevs
590K Followers 2 Following Official updates for developers building with @ClaudeAI
Lenny Rachitsky @lennysan
393K Followers 3K Following Deeply researched product, growth, and career advice
Blake Robert Mills @BlakeRobMills
2K Followers 442 Following Data Scientist ~ Cities + Statistics + Art ~ ABBA Enthusiast ~ #MetBrewer Creator ~ Cowboy at Heart ~ 🏳️🌈
SemiAnalysis @SemiAnalysis_
136K Followers 27 Following
Josh Lunger @joshjlunger
14 Followers 81 Following Computational neuroscience researcher @ Stanford, incoming PhD student in computer science. Building something new at South Park Commons.
Dwarkesh Patel @dwarkesh_sp
242K Followers 1K Following Host of @dwarkeshpodcast https://t.co/3SXlu7fy6N https://t.co/4DPAxODFYi https://t.co/hQfIWdM1Un
Dimitris Papailiopoul... @DimitrisPapail
28K Followers 1K Following Researcher @MSFTResearch, AI Frontiers | Prof @UWMadison (on leave) | babas of Inez Lily.
Blake Bordelon ☕️... @blake__bordelon
2K Followers 891 Following Postdoc @Harvard CMSA, starting UT Austin FL 2026. 2:48 marathon
Stanford AI Club @stanfordaiclub
2K Followers 22 Following Stanford’s premier student-led club focused on AI research and development.
Ludwig Schmidt @lschmidt3
7K Followers 426 Following Assistant professor at @Stanford and member of the technical staff at @AnthropicAI.
Francesco Cagnetta @Fraccagnetta
512 Followers 507 Following Theoretical physicist (bow tie included), inherently out of equilibrium. Studying data structure and deep learning. Marie Skłodowska-Curie fellow at @SISSA.
Wu Tsai Neurosciences... @StanfordBrain
11K Followers 412 Following Official feed of @Stanford Wu Tsai Neurosciences Institute. Unlocking the mysteries of brain and behavior through cross-disciplinary research.
Michelle Fang 🌁 @michelleefang
43K Followers 438 Following @stripe startups | ✍️ https://t.co/6ZpkX4oDDo
Tiberiu Mușat @Tiberiu_Musat_
781 Followers 955 Following Trying to figure out how AI works 🔍🧠 Currently at @ETH Zurich, previously @EPFL 🇨🇭 LLMs, interpretability, emergence, grokking 🤖
Ken Ono @KenOno691
67K Followers 200 Following Founding Mathematician @axiommathai | Prof @UVA | AI, proof | Assoc. Producer, The Man Who Knew Infinity | Olympic swim performance 🏊♂️🏅
Percy Liang @percyliang
110K Followers 425 Following professor of computer science @Stanford @stanfordnlp, co-founder of @togethercompute, creator of https://t.co/7R5THVogW2, co-founder of @simile_ai, pianist
Kazuki Irie @kzkirie
399 Followers 196 Following Computer scientist under @HandsomeDanYale. #KnowTogether @WuTsaiYale. KI, der KI erschafft. All tweets are wrong, but some are useful. No need to follow back!
Eric J. Michaud @ericjmichaud_
4K Followers 1K Following Trying to make deep neural networks among the best understood objects in the universe. 💻🤖🧠👽🔭🚀
Ziming Liu @ZimingLiu11
14K Followers 886 Following Assistant Professor @Tsinghua_Uni CollegeAI. Chief Scientist @MetaCircle_AI Science of AI, AI for AI, AI for Science, KAN.
Tianyuan Zhang @tianyuanzhang99
2K Followers 1K Following General intelligence and continue learning at meta tbd lab. prev Phd in MIT, M.S. in CMU, B.S. in PKU.
Sarah Catanzaro @sarahcat21
16K Followers 2K Following “All methods are sacred if they are internally necessary” (GP @amplifypartners, prev @canvasvc; Head of Data @Mattermark; @palantirtech; @c4ads)
Graham Neubig @gneubig
45K Followers 781 Following Associate professor @LTIatCMU. Co-founder/chief scientist @OpenHandsDev. I mostly work on modeling language.
Manasi Sharma @ ICML ... @ManasiSharma_
544 Followers 362 Following research engineer @scale_AI, working on reasoning for frontier models, agents, rl | prev @stanford, @StanfordAILab, @mitll, @Columbia
Lukas Muttenthaler @lukas_mut
839 Followers 739 Following Senior Researcher @AIgnostics & @ExplainableML | ex-@GoogleDeepMind | PhD in ML from @TUBerlin @bifoldberlin @MPI_CBS | Representations in 🧠 & 🤖| #FirstGen
Tatsuya Aoyama ✈️... @t_aoyam
89 Followers 344 Following Research Scientist @AIatMeta | Ph.D. @GUCompLing
Pratyush Maini @pratyushmaini
3K Followers 583 Following Data Quality x Memorization | Founding Team @datologyai | PhD @mldcmu | BTech @iitdelhi
Unconventional AI @unconvAI
8K Followers 3 Following Unconventional AI is rethinking the foundations of a computer to bring biology-scale efficiency to artificial intelligence.
Christopher Potts @ChrisGPotts
16K Followers 725 Following Stanford Professor of Linguistics and, by courtesy, of Computer Science. Member of technical staff @stanfordnlp and @StanfordAILab. Co-founder @ Bigspin AI.
Yizhou Liu @YizhouLiu0
813 Followers 196 Following PhD student at @MITMechE | Physics of AI, Physics of living systems, Complex systems, Statistical physics
Jyotirmai Singh @SinghJyotirmai
2K Followers 418 Following PhD'ing superconducting quantum @stanford physics, @ucberkeley | 🇮🇳🇸🇬🇦🇪🇺🇸 | sarvam idaṃ veditavyam
Francesco Orabona @bremen79
8K Followers 445 Following Dad and associate professor at @KAUST_News. Formerly @BU_ece, @sbucompsc, @YahooResearch, @TTIC_Connect. ML theory&practice, obsessed with history of scienceYou might like
