Pathway (www.pathway.com) @pathway_com

Last week, we sat down with some of San Francisco's most respected AI journalists for a conversation about where the field is headed, and what we at @Pathway believe are some of the most important questions: has the Transformer architecture underlying every major model finally hit its limits? And how can we reliably improve the intelligence per dollar ratio? We spent much of the night talking about the problems AI faces today, namely (1) that thinking and problem-solving are only add-ons to the Transformer, designed initially to process language / sequences at scale (2) and that the Transformer does not support continual learning, native persistent memory, nor experience-grounded reasoning. The cracks are evident, with the ability to solve tasks longer than a Transformer’s context being still both limited, patchy, and incredibly costly. Thank you to all those who came and made it such a wonderful evening! Martin Farach-Colton @nyutandon @adrian_pathway Also PSA for San Francisco if you have not been to Flour + Water, it is amazing!

keyboard_arrow_left Previous keyboard_arrow_right Next

@zuzanna_pathway @adrian_pathway For the next generation of changemakers!

Excited to join the Bison Fellowship as mentors alongside @adrian_pathway. We’re supporting Poland’s brightest young AI and science talent as they build ambitious ventures with AI and spend time in San Francisco connecting with the ecosystem here. Looking forward to meeting this year’s fellows 🇺🇲 Big thanks to @kawecki_maciej for creating this program.

Last week’s Post-Transformer debate post raised one question: Can long term memory become part of the architecture? It points to one promising mathematical idea behind Post Transformer AI: Linear attention in high dimension with persistent state. In a standard Transformer, memory is handled through caching context. The model keeps previous keys and values in small dimension d, then attends over them. But this is still token history. BDH (Dragon Hatchling) – one of the Post-Transformer architectures, takes a different route. The paper describes BDH's state space as fixed and large, with the macro interpretation of associative memory, like KV cache, but organized differently. Each layer has a persistent state matrix: ρₗ ∈ Rⁿˣᵈ Here: n = neuronal or concept dimension d = low rank synaptic dimension d << n The key idea is that state is aligned to neurons, in high dimensional space (n in the order of billions). A Transformer stores token history.Whereas BDH-GPU (a tensor-friendly version of the BDH architecture) evolves state, similar to State-Space Models. This is where the brain analogy becomes useful. The brain does not append every experience into a longer transcript. It has a large bounded substrate of neurons and synapses, where experience changes connections sparsely and with high parallelism. BDH GPU expresses a related idea computationally: not memory as a longer context window, but memory as a large, evolving internal state. Why it matters: – no Transformer style hard context window. practically enabling a infinite context window in a reasoning model. – linear attention in a large neuronal dimension – sparse positive activations – persistent state instead of only token history The deeper insight: Long horizon reasoning may not come from storing more tokens. It may very well come from better state dynamics.

Exactly why I’m so bullish on @pathway_com. 🧠 Next-Gen AI ⚡️ Incredible performance 🔋 Ultra compute-efficient

Thierry from arvy 🇨🇭 @ThierryBorgeat

4 weeks ago

Citadel Securities just put institutional weight behind what the AI bulls won't say out loud. In a new macro note titled "Tokenomics," Citadel makes the argument plainly: even the most powerful technology on earth still has to pass through the boring discipline of cost curves,

118 502 3K 492K 3K

There are co-founders. Then, there are friends. And then, there is @JChorowski. Jan Chorowski and his story. From working on attention at MILA, through speech to Google Brain, all the way to BDH.

@zuzanna_pathway @JChorowski From early attention research to building the Post-Transformer era for the world! @JChorowski🫡

@oraclekev Great pick, @oraclekev! You might also like Łukasz Kaiser in this debate with his peer inventors of Transformer and Post-Transformer architectures: youtube.com/watch?v=hCjoML…

This is probably the most entertaining way to understand one of AI’s hardest AI debates. Transformer vs Post-Transformer, argued by leading researchers, inside a real physical boxing ring. Both technically deep and genuinely entertaining. I was glued for the entire 1 hour 20 minutes. So many super cool points to learn. 🥊 Transformers - Transformers still own the present because they work at scale. They are simple, trainable, hardware-friendly, and already power the strongest AI systems we use today. - The Transformer is basically a memory machine. It stores information as keys and values, then uses attention to pull back the most useful parts when answering. - The real Transformer advantage is not just “attention.” The bigger advantage is that it fits modern hardware extremely well, so it can process huge batches of tokens fast. - Scaling is still the brutal rule. If you give Transformers more compute, more data, and more parameters, they usually keep getting better. Any Post-Transformer architecture has to scale just as well, or better. - It is not enough to look clever on small tests, because the real question is whether it improves faster than Transformers when scaled up. - A replacement cannot be slightly better. Because the whole AI stack is already built around Transformers, the next architecture may need to be around 10x better to force everyone to switch. - Transformers are powerful, but they may be brute force. A human does not need to read the entire internet many times to become smart, but current LLMs need enormous data and compute. 🥊 Post-Transformer - Post-Transformer people are not saying Transformers are bad. They are saying Transformers may be the best current tool, not the final form of machine intelligence. - The biggest Post-Transformer target is native reasoning and continual learning. Today’s LLM reasoning often feels like text-based step-by-step work added on top, instead of thinking happening naturally inside the model. - Latent reasoning is one possible next step. That means the model reasons inside its own hidden internal space, instead of writing every thought out as words. - Continual learning is still a major weakness. Humans keep learning from experience, but most Transformer-based models are trained, frozen, and then only adapt inside the prompt. - Long context is not the same as real memory. A model can read a huge prompt, but that is different from building a life history, learning from mistakes, and updating beliefs over time. - The future may be hybrid, not a clean replacement. Transformers may stay as 1 building block while newer systems add better memory, better reasoning, and better learning loops. - The most interesting possibility is that Transformers may help discover their own successor. AI agents are already getting better at research and coding, so the next architecture may come from AI-assisted architecture search. ------- - Benchmarks are a problem. Many public benchmarks are easy to game, so they may show leaderboard strength without proving deeper intelligence. - Perplexity is still probably a great metric to evaluate frontier models,, because it tests prediction quality. --- Overall, Transformers continue to dominate, but the frontier is clearly widening. Pathway’s BDH (Dragon Hatchling — brain-inspired reasoning architecture), Sakana AI’s CTMs (Continuous Thought Machines — models that think over time), and Liquid AI’s LFMs (Liquid Foundation Models — efficient multimodal foundation models) - all of these show how the frontier is expanding. --- From “Pathway (pathway[.]com)” Youtube channel (link in comment) @zuzanna_pathway

200K+ views and counting! The Transformer vs Post-Transformer debate, convened by @pathway_com Ft @lukaszkaiser, @adrian_pathway, @YesThisIsLion, @mlech26l, @dexhorthy, and me. Watch on YouTube: youtube.com/watch?v=hCjoML… Follow along for the next one.

@zuzanna_pathway @lukaszkaiser @adrian_pathway @YesThisIsLion @mlech26l @dexhorthy First of many! 🐉🔥

“We have not yet had a PageRank moment for intelligence.” We’ve got so many comments and questions about this statement delivered by @adrian_pathway during our recent Transformer vs Post-Transformer debate with @lukaszkaiser @YesThisIsLion @mlech26l - thanks! Let’s dig into it. In the 1990s, web search already existed. We could index information. AltaVista existed. The web was growing fast. Then PageRank happened. That moment combined three things: 1. A simple but deep mathematical idea: treat the web as a giant graph and compute a stationary distribution of a *random walk* on that *graph* 2. A scalable implementation: large-scale graph computation on huge clusters 3. A company that integrated and scaled the idea end-to-end: Google That combination gave search a much clearer center. It stopped being just a pile of heuristics and started to look more like: here is the mathematical object we need to compute, now let’s build the systems needed to compute it well. Adrian asked Lukasz Kaiser directly whether he sees a PageRank-like idea inside the Transformer. Lukasz said no. For intelligence, we still do not have that kind of unifying operator or process. We do not yet have an agreed mathematical object that says: this is the core computation behind it. That missing unifier is what Adrian meant by the absent “PageRank moment for intelligence.” That is also the main idea behind our work on BDH, our Post-Transformer architecture. We are after that fundamental “platform discovery” for intelligence. The full Transformer vs Post-Transformer debate is a good place to go deeper on these topics. Link below.

@rohanpaul_ai Thanks for the sharing your key observations from the debate, @rohanpaul_ai! We're glad you found it useful!

Here's a great starting point for you to understand the Transformer vs Post Transformer Debate convened by @zuzanna_pathway! Credits @rohanpaul_ai.

Rohan Paul @rohanpaul_ai

a month ago

This is probably the most entertaining way to understand one of AI’s hardest AI debates. Transformer vs Post-Transformer, argued by leading researchers, inside a real physical boxing ring. Both technically deep and genuinely entertaining. I was glued for the entire 1 hour 20

7 24 109 89K 101

@probnstat Thanks for sharing, @probnstat! This is quite nicely put. It was one of the threads briefly covered in the Post-Transformer debate with the inventors behind these architectures. Worth a watch: youtube.com/watch?v=hCjoML…

The full Transformer vs Post-Transformer debate is live. 80 minutes. Seven rounds. No slides. Real disagreement. @lukaszkaiser came to defend the Transformer. @adrian_pathway, @YesThisIsLion, and @mlech26l made the case for what comes next. 00:00 Contenders enter the ring 06:30 Lukasz Kaiser defends the Transformer 10:08 Adrian Kosowski on BDH and the PageRank Moment for AI 17:35 Llion Jones: Why Transformers aren't the final architecture 29:50 Mathias Lechner on Liquid AI’s approach, Fast Weights, and Self-Replacing AI 40:28 Reasoning Beyond Language 44:15 Scaling Laws: Transformer vs Post Transformer 50:31 Benchmarks, Coding Models, and Perplexity 1:04:00 Continual Learning and Dynamic Weights This is the ultimate source of truth on the subject.

@zuzanna_pathway @AWSstartups Reasoning ➕ Long-term memory = your context, compounding.

@probnstat Appreciate the writeup, @probnstat! The 80 minutes flew by because every one of those open questions deserves its own deep dive. Well, Post-Transformer era is here, and we'll have lots of opportunities in the near future.

One deep learning debate every AI researcher should care about: Transformers vs Post Transformers. At the surface, it sounds like an architecture fight. Mathematically, it is about scaling laws, memory, online learning in frontier models, and hardware limits. That is what made the recent debate interesting. It featured @lukaszkaiser, @adrian_pathway, @YesThisIsLion, and @mlech26l, hosted by @zuzanna_pathway. Transformers won the last era because multi head self attention scales empirically and fits the hardware ecosystem extremely well. But the next bottleneck may be different. Full self attention has O(n²) compute pressure with sequence length. Transformer LLMs do not natively have persistent long-term memory. RAG retrieves. Longer context conditions. Neither necessarily forms new reasoning patterns inside the model. That is why continual learning is becoming central, recently covered by @a16z. The open questions: – How can models learn after deployment without catastrophic forgetting? – How can long term memory become part of the architecture? – How can models reason over longer horizons without paying infinite context costs? – How can hardware and AI architectures co-evolve more efficiently? – And, are we chasing the right benchmarks with these goals in mind? These questions were tackled head on, with counters from @lukaszkaiser, Transformer co-inventor and core contributor to ChatGPT and GPT models. The image below summarizes some notes from the 80 minute debate.

先日サンフランシスコで開催された討論会「Transformers vs Post-Transformers」に、Sakana AIの共同創業者兼CTOであるLlion Jones @YesThisIsLion が登壇しました。 本イベントは、現在のAI界を牽引するアーキテクチャ「トランスフォーマー」について、論文共著者を含む4人が、トランスフォーマー 支持と、継続学習や潜在空間での推論を武器にその次を見据える「ポスト・トランスフォーマー」の支持に分かれ、これからのAIの未来をどちらが形作るのかを深く議論する場となりました。 その中でLlionは、トランスフォーマーの原論文の共著者でありながら、現在のトランスフォーマーの有用性は十分に認めつつも、あえてポスト・トランスフォーマー側に立ち、その先のアーキテクチャの可能性を論じる役割を担いました。 Llionは、現在のトランスフォーマーの成功は構造そのものによるものではなく、並列処理に優れたハードウェア（GPU/TPU）に適応できたことによる「計算資源の力技」による側面が大きいと分析。それと並行して全く異なる前提に立つアーキテクチャを探る重要性を提起しました。 さらに、今後の研究コミュニティに対して、既存のベンチマークや現在のハードウェアの制約から解放されるべきだと提唱。「次の革新的なアーキテクチャは、初期段階ではトランスフォーマーより遅く、精度も劣るかもしれない。しかし、それを恐れずに全く異なる前提のシステムを探求すべきだ」と、研究姿勢そのものの変革を訴えました。 Sakana AIはトランスフォーマーをベースとした研究開発と並行して、次世代アーキテクチャの探求にも研究にも取り組んでおり、Llion自身が関わっている、生物学的な脳に倣った新アーキテクチャであるContinous Thought Machine（CTM）などはその一例です。 刺激的な議論の場を提供してくださった主催者の皆様、そして登壇者の皆様に心より感謝申し上げます。 当日の討論会の様子は、こちらからご覧いただけます： x.com/zuzanna_pathwa… 🐟 @zuzanna_pathway

Zuzanna Stamirowska @zuzanna_pathway

2 months ago

The full Transformer vs Post-Transformer debate is live. 80 minutes. Seven rounds. No slides. Real disagreement. @lukaszkaiser came to defend the Transformer. @adrian_pathway, @YesThisIsLion, and @mlech26l made the case for what comes next. 00:00 Contenders enter the ring 06:30

17 21 214 1.4M 139