Seungwook Kim @1ndependentGrad

Umar just made a super long 19-hour video explaining distributed training from first principles (while visiting concepts like RoPE, KV Cache, MLA, etc.) I've been missing his tutorials ever since he joined Mistral, one of the best educators on YouTube. So happy he's back.

Umar Jamil @hkproj

3 days ago

In this (extremely long) video, I'll build a distributed training framework from first principles using PyTorch. We'll derive everything from first principles, explaining the mathematics behind distributed training as well as every auxiliary component we build along the way,

84 252 2K 293K 3K

KVキャッシュの考え方を視覚的に理解しやすい形でブログにしてみました。よければご覧ください！ TransformerのKVキャッシュを用いたDecode処理を理解する qiita.com/kenmatsu4/item… #Qiita @Kenmatsu4より

keyboard_arrow_left Previous keyboard_arrow_right Next

この論文はYann LeCunのLeWorld model論文と合わせて読むと面白いと思うんだけど、あまり日本国内では話題にならなかった感じなんだろうか arxiv.org/abs/2605.27734

Kimi K3's MoEs should have collapsed. 🫠 But it didn't. How did they stabilize training for highly sparse MoEs? The key is Stable LatentMoE: • Normalization on the routed branch • SiTU-GLU inside each expert • Quantile Balancing on the router Learn more here: youtu.be/vXV2ltVdLMQ

Google just released a free 2-hour course on graph engineering and it's the clearest breakdown of wiring agents into a system that scales on its own. 17:44 - your first AI agent 39:30 - running agents with loops 1:12:38 - turning loops into graphs 1:34:26 - self-improving agents 1:55:05 - a complete graph that runs itself Worth more than any $700 course you'll find. Watch it, then go further with the step-by-step guide below on how to build a system that improves itself.

Anatoli Kopadze @AnatoliKopadze

4 weeks ago

x.com/i/article/2080…

77 419 3K 9.5M 13K

Ilya Sutskever 认为吃透30篇AI奠基论文，就能掌握人工智能绝大多数核心精髓。最大难点在于看得懂公式，却难以落地成代码。 开源项目 Sutskever 30 补齐了这一环：纯NumPy从零实现全部30篇经典论文，不依赖深度学习框架，Notebook 直接运行。从循环网络、Transformer 到 VAE、神经图灵机全部覆盖。 适合想要弄懂底层逻辑，拒绝只会调用API的开发者。 github.com/pageman/sutske…

Most Transformer interview questions sound easy until someone asks you to explain what’s actually happening under the hood. > Why divide attention scores by √dk? > Why can training be parallel while autoregressive decoding can’t? > Why does GQA reduce inference cost? > What exactly is quadratic in self-attention? > Why does Pre-Norm train differently from Post-Norm? I put together 25 Transformer architecture interview questions covering the mechanics behind attention, position, normalization, and sparse computation. ➡️ Attention → What Transformers changed vs. recurrence and convolution → Q/K/V and scaled dot-product attention → Self-attention vs. cross-attention → Causal masking: parallel training vs. sequential decoding → Multi-head attention, and why individual heads resist clean interpretation → Tensor shapes: dmodel, heads, dhead, dff → MHA vs. MQA vs. GQA and the KV-cache tradeoff → What actually scales quadratically → FlashAttention and why reducing memory movement matters → Sliding-window attention and local + global attention ➡️ Position → Why attention alone has no built-in token order → Absolute vs. relative positional methods → RoPE and why queries and keys are rotated → How RoPE makes attention depend on relative displacement → ALiBi ➡️ Representations & stability → Input embeddings, LM head, and weight tying → Residual connections → RMSNorm vs. LayerNorm → Pre-Norm vs. Post-Norm ➡️ FFNs & sparse computation → What the FFN contributes beyond attention → ReLU/GELU vs. gated FFNs such as SwiGLU → Why the original Transformer used a much wider FFN hidden dimension → Sparse Mixture-of-Experts, routing, and active vs. total parameters I also added interviewer notes for 8 of the trickier questions: follow-up probes, common weak answers, and red flags. Plus a section on attention sinks and a few details that are often skipped in basic Transformer explainers. Every answer is grounded in the original papers, including work from Vaswani, Shazeer, Ainslie, Dao, Beltagy, Su, Press, Zhang, Xiong, Fedus, and Xiao.

Tech with Mak @techNmak

5 days ago

A curated list of 25 LLM foundations interview questions. What's covered Core mechanics: → What happens from input text to first output token → Autoregressive modeling and chain-rule decomposition → Logits, softmax, and how probabilities are formed Decoding & training

10 38 238 59K 405

Keeping up with @arxiv is impossible. 500+ preprints a day across dozens of fields. @KurateOrg scores papers on 16 dimensions (impact, novelty, rigor, surprise, translational...) so you can spot what's worth reading at a glance. Sort, filter & sign up for the weekly newsletter.

Why Kimi K3 doesn’t need RoPE RoPE encodes relative position info: for any positions m and n, we need the transformations on query and key to cancel out into a relative position after doing q @ k.T Math: R(m).T @ R(n) = R(n-m) It is proven that any powers of orthogonal matrix (A^m) can be RoPE Householder matrix, a type of matrix, satisfies the requirements If you replace RoPE with Householder matrices, after 🧑‍🍳 hardcore math🧑‍🍳, DeltaNet pops out in the softmax attention formula Crazy, I know. DeltaNet literally has a Householder matrix in it though Kimi Delta Attention is a DeltaNet variant Hence, KDA can replace RoPE

a16z @a16z

2 weeks ago

vLLM runs on half a million GPUs at any given moment. Most people have never heard of it. Simon Mo, co-founder and CEO of @inferact and lead maintainer of vLLM, sits down with a16z’s Matt Bornstein and Elena Burger to discuss what it takes to actually run open models in

11 51 339 209K 348

NVIDIA researchers did it again! They found a way to make KV cache transferable between models. The target model skips prefill entirely, and the conversion runs 2.7 to 25x faster than processing the context again. Let's understand why this is so important today. LLM APIs are stateless, so every turn sends the entire conversation back to the model. The model reads all of it again before writing a single new token, and all of it is billed as input. Prompt caching allows Anthropic and other providers to hold the KV cache for a stable prefix and bill a hit at roughly 10% of the base input rate, because the compute was already done once. The 90% reduction is one of the largest lever in LLM serving, which is why so much production work goes into keeping prefixes byte-stable. But the cache only works on the model that produced it. Keys and values are produced from that model's weights, so no other model can read them. In pratice, the constraint shows up in LLM routing. If the traffic is shifted to a different model for cost/capability reasons, the accumulated KV cache becomes invalid. As a result, the accumulated context has to be processed from scratch, and it's billed at full rate. NVIDIA's recent paper treats this as a representation problem. Prefill's only output is the KV cache, so to move KV between models, we need to convert one model's cache into the format the other expects. They first checked whether the conversion has any structure worth exploiting. They found that moving from Qwen3 14B to 32B, a plain linear regression from a single source layer reconstructed 56% of the variance in the target model's keys. The two models obviously may have different layer counts, so there is no natural one-to-one pairing between them. For each target layer they rank every source layer by how well it predicts that layer, then feed the top eight in together, which takes the reconstruction to 79%. The mapper itself has three parts: > Each target layer and head gets its own independent linear map, solved in one closed-form step rather than by gradient descent. > The cross-layer selection described above is the second part, and their ablation shows it carries the most weight of the three. > Keys also carry a position-dependent rotation from RoPE. They strip that rotation, fit the map in position-free space, then re-apply the target model's rotation at inference. Across six pairs from Qwen3, Llama 3.1 and Ministral 3, four retain 73 to 98% of the receiving model's standalone accuracy, and the conversion runs 3-25x faster than processing the context again. Prior work on cross-model KV reuse exists, but it either trains a neural adapter per pair or requires both models to be architecturally identical. This is probably the first version that is closed-form and training-free, so a lot of it is still open research. Every pair tested belongs to one family, so it works on Qwen to Qwen and Llama to Llama. Cross-family transfer is listed as future work. All six pairs mentioned above also happen to share KV head count and per-head dimension across scales. Mismatched head configurations are currently untested. The researchers scoped this to dense full-attention only, so sliding-window and attention-recurrent hybrids still need work. Here's the paper: arxiv.org/abs/2608.03893 Plenty of work is yet to be done. Still, the constraint being solved is genuine. Every model swap currently invalidates the full KV that was already paid for, and this is the first result showing that work might be recoverable without training anything extra. That said, all of this only matters because of what the KV cache is doing in the first place. I wrote a first-principles breakdown of it, covering why the model stores keys and values at all, why the cache grows with every token, and what generation speed looks like with and without it. Read it below.

Avi Chawla @_avichawla

5 months ago

x.com/i/article/2034…

33 468 3K 1.8M 6K

Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks also to everyone who shared ideas and opened PRs with improvements! Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories). I am also currently working on a larger applied custom “small” LLM project. It has been keeping me super busy this month, but I will share more on that soon. If you are new to it, some of the highlights include 1. Of course, the complete code path from tokenization and attention to pretraining, classification, and instruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.) 2. From-scratch implementations of Llama, Qwen, Gemma, and Olmo (smaller variants that run locally and can be plugged into the training scripts). 3. From-scratch implementations of attention alternatives and other architecture components, such as GQA, MLA, sliding-window attention, Gated DeltaNet, DeepSeek Sparse Attention, cross-layer KV sharing, and mixture-of-experts 4. Materials on KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA So, if you don’t have any weekend plans yet, happy tinkering!

分享一套很适合补「现代强化学习」的开源教程：Hands-On Modern RL。 它的学习路线很完整，从 CartPole、DQN、PPO 这些经典 RL，一直讲到 RLHF、DPO、GRPO、RLVR，再往后延伸到 Agentic RL、工具调用、多轮信用分配和 VLM RL。 我比较喜欢它的一点，是「实践优先」。很多章节都配了可以直接运行的代码、训练曲线和失败案例，比如从零实现 PPO、数学推理 GRPO、Mini Deep Research Agent，以及 VLM 的 GRPO 训练。训练崩溃、Reward Hacking、KL 漂移这些实际问题也被当成正式内容来讲。 尤其 Agentic RL 部分很值得看。它把训练对象从一次回答扩展成完整 trajectory，讨论工具调用、环境反馈、多轮信用分配和异步 rollout，而这些正是 Agent 从「会回答」走向「会长期行动」之后绕不开的问题。 现在很多 RL 教程还停留在经典控制，这套资源已经把学习路线一路接到了今天的大模型后训练和 Agent 研究。 不过项目自己也提醒，目前仍在快速迭代，而且有 AI 协助生成，部分内容尚未全面审稿，阅读时最好结合论文和代码一起验证。 项目链接： github.com/walkinglabs/ha…

🚀Excited to share Qwen-3D (ECCV 2026) — a generalist 3D vision–language model that does grounding, segmentation, VQA, and spatial reasoning in one place. VLMs are great at images and short clips, but long videos bury them in tokens. Our idea: put multi-view frames into a shared 3D world, so attention scales with what’s in the scene, not how many frames you feed it. 📄arxiv.org/abs/2608.02980 🌐qwen-3d.github.io

Really enjoyed this paper: “Is One Layer Enough?” A surprisingly simple finding: you may not need to update the whole LLM during RL post-training. Training just a single Transformer layer can recover most of the gains from full-parameter RL — and sometimes even outperform it. Even more interesting, the effect is highly structured: 🧠 High-contribution layers consistently concentrate around the middle 40–60% of the network 📈 The pattern holds across 7 models, 3 RL algorithms, and math/code/agentic tasks 🔄 Layer rankings remain correlated across datasets and tasks 🚀 Using this structure for layer-aware RL can outperform full-parameter training This feels like a really interesting clue about where RL actually changes an LLM — and suggests that post-training may be much more localized than we usually assume. Very nice work! arxiv.org/pdf/2607.01232

keyboard_arrow_left Previous keyboard_arrow_right Next

A good view at emerging theory on deep learning arxiv.org/abs/2604.21691

We live in a multimodal world. We see, talk, act, and dream. Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch? Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition So what’s the path forward? Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects/physi…) We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

Very elegant idea! The best kind of work has the simplest pseudocode. Also ties into ideas from evolution about maintaining diversity via exploration (i.e. you want enough exploration such that at least one of your mutations is good).

Alexi Gladstone @AlexiGlad

3 weeks ago

We discovered a third pretraining axis beyond parameters and data: exploration. Scaling exploration monotonically improves existing models across images/video/language, and unlocks end-to-end generation. In the simplest case, it's just a for loop. Introducing Explorative

113 317 3K 1.1M 3K

这篇普林斯顿博士论文，几乎把强化学习过去十年的两条主线串在了一起： 从博弈算法，一路走到世界模型。 作者 Zihan Ding，导师是 Chi Jin。整篇论文超过 300 页，汇集了博士期间八项研究工作，其中多项成果发表于 NeurIPS、ICML、ICLR 和 ICCV。 前半部分研究多智能体博弈：如何学习接近纳什均衡、难以被对手利用的策略，并把这些方法扩展到格斗游戏、多人博弈和真实网络负载均衡。 后半部分转向 Foundation Model 时代：扩散世界模型、生成模型策略、少步视频生成、可交互视频世界模型，以及支持长时预测的记忆机制。 这篇论文最有价值的地方是它记录了强化学习研究重心的一次迁移。 过去，RL 更关注如何在给定环境中学会行动。 现在，Agent 还需要借助生成模型建立对环境的理解，预测不同动作会带来什么未来，再据此规划和决策。 这里收录的也不只是零散的想法，而是一条经过多年研究和多篇顶会论文逐步铺开的路线。 很多论文告诉你某个方法如何提升一个指标。 这篇博士论文更像一张地图，让人看清强化学习如何从策略优化，走向对复杂世界的建模。 想系统理解 RL、世界模型和生成式 AI 之间的关系，这篇很值得完整读一遍。 📎 arxiv: arxiv.org/abs/2607.17560

あっー！ 神神神神神神 Qwen 3あたりで知識止まってたので本当にありがとう！！ マジで！！！ありがとうすぎる！！！！！！ みんなも、読めよな

ali @waterloo_intern

3 weeks ago

x.com/i/article/2077…

237 3K 14K 6.5M 31K

WAIT... WHAT!! Anthropic burned $3 million and 30 months running AI agents against real tasks in real companies and published everything in 12 pages and the people who read those 12 pages this week are going to make decisions about how they build and work that are fundamentally different from the people who scroll past it thinking they already understand what AI can do. 1,287 real tasks. 2.4 million lines of code. AI outperforming humans on their own projects. 63% of routine work automated. Engineer output multiplied three times with the right memory and the right cycles. Someone paid $3 million so you could read that for free and most people are going to bookmark it and never open it again which is the most expensive free resource in the history of AI research. But here is the gap that the 12 pages do not close. Knowing that the right system multiplies output three times and knowing how to build that system are two completely different things and the second one is where everyone gets stuck and stays stuck while the people who figured it out quietly operate at a different level. My article is the bridge between the $3 million research and the actual system build and it is the thing you read immediately after you bookmark those 12 pages. Do not let this be another thing you meant to act on. Full guide below.

Kanika @KanikaBK

4 weeks ago

x.com/i/article/2080…

2 44 269 572K 1K