Swetava Ganguli @Swetava

Releasing the model weights and technical report of Kimi K3. Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window. New model architecture: 2.5x the intelligence per unit of compute, not just more params. Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale. Model weights: huggingface.co/moonshotai/Kim… Tech report: github.com/MoonshotAI/Kim… Tech blog: kimi.com/blog/kimi-k3

It is a hard and sad decision. I shared this message with folks at Thinky. Thank you all for the time together♥️ Just as the last sentence in my message: The future worth building is human.

Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run AI on our own hardware if we are not ready to share personal data and IPs with closed labs through using their models. (Not that proprietary models are bad, actually I use them a lot as well, but it wouldn't healthy not to have any alternatives.) Anyway, while pretty much everyone is waiting for the Kimi K3 and Ling 3.0 weights to land on the model hub any day now, there were quite a few other interesting new open-weight model releases the past week. Yes, one of those weeks! So, here are the architecture pics along with some notes on what I found most interesting: 1) Nanbeige 4.2 3B uses looped depth sharing. This basically means it runs the same 22-layer (=transformer block) stack twice. So, it extends the 22-layer architecture to 44-layers, but without duplicating the weights. (2x the transformer block compute but same memory footprint.) Why? The info is a bit sparse, but section 2.1 of the Nanbeige 4.2 technical report says two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made the training much slower and much more expensive. 2) Laguna S 2.1 is poolside's Laguna model in a really nice size: 118B sparse MoE with 8B active parameters and a 1M-token context window. Otherwise, the architecture is pretty standard. It uses 36 sliding-window and 12 global (gated-)GQA layers. However, given this size, and the fact that it (just barely) runs on my DGX Spark (uses about <80 GB of RAM), this is right now the most interesting model for me personally. It's 3x bigger and thus a tad slower but maybe a good candidate as daily-driver-Qwen3.6-35B-replacement. (Still waiting on some more independent performance benchmarks though.) 3) Motif-3-Beta is a new 314B-A13B sparse MoE that is somewhat based on DeepSeek V4 in terms of mHC and latent attention. But it uses a new component, Grouped Differential Latent Attention, which is inspired by Multi-head Latent Attention. I probably should write an article about this some time, but for now, the tl;dr is as follows. Regular MLA compresses the keys and values into a smaller latent representation to mainly reduce the KV cache size. GDLA does a similar low-rank compression but puts the attention heads into groups and also learns a noise head for each group where the noise gets subtracted for filtering purposes... Anyway, a topic for another day! 4) Solar Open 2 is a new 250B-A15B hybrid MoE by Upstage that interleaves three Kimi Delta Attention layers with one GQA layer. 5) Antares 1B is a small model (and there is also an even smaller 0.3B variant) from Cisco starts that with the IBM Granite 4.0 1B backbone and uses SFT plus GRPO for terminal-based cybersecurity stuff. It is a nice example of task-specific post-training on a genuinely small model. 6) BTL-3 is a rank-32 LoRA adapter for Qwen3.6-27B aimed at coding agents and structured tool use. The really strong benchmark performance suggests that LoRA adapters are still a useful tool/technique in 2026. I added all six to the LLM Architecture Gallery for some additional details: sebastianraschka.com/llm-architectu…

great resource just published from @Modular!

New lecture! This one is a recap of a bunch of history of preferences, the nature of rewards, how RLHF is formulated, which were once seen as central problems in the field. How much as changed. Still... super interesting to understand our optimization tools today. Books coming soon :D 00:00 Intro & context 07:34 A short history of preferences (from Aristotle to the VNM Utility Theorem) 20:17 A brief overview of preference data (from the last two years of my practice) 31:11 Open questions in RLHF data Lecture 8, covering Chapters 10 & 11 of my book.

A great resource on PyTorch Internals

voxium @v0xium

2 weeks ago

I implemented Andrej Karpathy's Micrograd from scratch in C. In this video, we'll go through: • Forward & Backward pass of Add, Sub, Mul • Traversing the Computational graph using DFS/Toposort • Automatic differentiation If you've ever wanted to understand what PyTorch's

3 13 121 18K 102

Prediction is compression. Same thing.

fintex @_yusufknl

2 weeks ago

As someone who's been shipping LLMs since the GPT-2 days, this lecture on cross-entropy from a Stanford math grad is the closest thing to an ML PhD qualifying exam I've ever seen released publicly for free. Everyone thinks language models predict the next word. They don't. They

69 311 4K 350K 8K

My book, Reinforcement Learning from Human Feedback is done! This is the book I wish I had when learning to fine-tune, align, & now post-train models since ChatGPT. The resource has been built by me finding time to study and document the fundamentals on nights and weekends since 2024. Transferring as much of the intuitions of building Olmo as I possibly can in the book format. The book is launching with an over 10 hour, full course with slidedecks, functional code for the training chapters, an example model completions library, and of course the free online web version. Physical orders from Manning will ship in 1-2 weeks, and Amazon a week or so after. Thanks for your support!

90% of your KV cache never gets reused. (prompt caching was never meant to fix it) if your system prompt and tool definitions are stable, prompt caching is the single highest-leverage optimization available today. cached input tokens get up to 90% cheaper, and hit rates of 60 to 85% are realistic. but it comes with one rigid rule. the cached portion must be an exact, byte-for-byte prefix of the new request. change a single character in that region and you get a full cache miss. that rule breaks in three situations you hit constantly: → RAG with multiple documents. you cached document A alone and document B alone. a query now needs both. document B's cached state was computed without any awareness of A, so it's invalid and gets recomputed from scratch. → document order changes. the same three documents appear in a different order across requests. every permutation is a cache miss, even though the content is identical. → growing conversation history. each new turn changes everything after the stable prefix, so earlier cached states beyond it become useless. Alibaba Cloud's production data shows how bad this gets: 10% of KV cache blocks serve 77% of all hits. the rest sits in storage and never gets reused, because prefix matching won't allow it. CacheBlend, a research paper from the LMCache team (EuroSys 2025 Best Paper Award), attacks exactly this. the insight is that in modern transformers, tokens overwhelmingly attend to their own local context. only a small fraction of tokens carry real connections across document boundaries. so instead of recomputing everything after the first cached document, CacheBlend reuses every document's cache as-is and selectively recomputes just those few boundary tokens. those are the small orange fixes between documents in the diagram, and they are the entire cost. the result is 2 to 4x faster processing on multi-document queries with no quality loss. the order problem disappears with it. shuffle the same documents however you like, and every permutation stays cached, where prefix caching recomputes all of them every time. the bottom of the diagram shows that side by side. that's the real shift: from caching prefixes to caching knowledge. every document in your knowledge base becomes a reusable cached asset, regardless of what order it appears in or what sits next to it. CacheBlend ships inside LMCache, the open-source cache management layer that runs outside the inference engine and integrates with vLLM, SGLang, and TensorRT-LLM, on both NVIDIA and AMD GPUs. check it out on GitHub: github.com/LMCache/LMCache (don't forget to star 🌟) i wrote the full breakdown of the architecture, including why cache management should never live inside your inference engine. the article is quoted below. stay tuned for more on this!

Akshay 🚀 @akshay_pachaar

4 weeks ago

x.com/i/article/2074…

15 36 233 478K 418

Google engineer explained how to fine-tune a tiny LLM from 46% to 90% accuracy on your phone in 21 minutes - better than $1500 on-device AI bootcamps. pick Gemma 270M -> generate synthetic task data -> fine-tune with LoRA -> quantize to int4 -> deploy to Pixel and hit 2000 tokens per second. That loop is how a 270M model beats a 70B one on your task, running fully offline in your pocket. Gemma 270M + synthetic data + LoRA + int4 quantization + on-device runtime - that's the stack. Watch and save it, then fine-tune your own tiny agent tonight.

h100envy @h100envy

3 weeks ago

x.com/i/article/2077…

19 58 455 1.0M 1K

Stanford dropped their latest course on Parallel Programming, GPU, and CUDA. 24 hours, 19 lessons. this is one of the hottest skills that AI labs are looking for. it covers: > GPU architecture and CUDA > performance optimization > multi-core processors and architectures watch here: youtube.com/playlist?list=…

An Ex-Meta L8’s Agentic Engineering Setup In this guest article, @kunchenguid shares the agentic engineering workflow he uses on a day-to-day basis. Read the full article here: blog.bytebytego.com/p/an-ex-meta-l…

CMU Advanced NLP Lecture 9: Decoding Algorithms This lecture explains a key aspect of generative LLMs: The model learns a probability distribution, but useful generation still depends on how we decode from that distribution. 🔹 Greedy decoding picks the most likely token each step, but local best choices may not produce the best full sequence. 🔹 Beam search keeps multiple candidate paths, making decoding closer to sequence-level optimization. 🔹 Sampling turns probabilities into diverse outputs, but naive sampling can become incoherent because of long-tail tokens. 🔹 Top-k, top-p, and temperature control the tradeoff between quality, diversity, and randomness. The key idea: LLM generation is not just “the model predicts words.” It is model probabilities + decoding strategy. My note: ickma2311.github.io/ML/NLP/cmu-adv…

RL Algorithm Interview Questions 2026 (as compiled by @sheriyuo) k-a.in/rl-algo.html

Jane Street pays $650,000 a year for quants. Stanford just released the exact RL-for-trading bible for free. 16 chapters. 0 to algo trader. Asset allocation, market making, American option exercise, full Python code & Colab notebooks. Bookmark & give it a weekend.

This 115-page book unlocks the secrets of LLM fine tuning. drive.google.com/file/d/1cS5sWZ… A comprehensive guide which covers: > the fine-tuning process for LLMs > combining both theory and practice.

The Hands-on Modern RL tutorial everyone has been waiting for is finally available in English🥳🥳🥳 PDF download link: github.com/walkinglabs/ha…

Sanbu 散步 @sanbuphy

3 months ago

Today we released the English version of Hands-On Modern RL along with a downloadable PDF, fully open and free. The course spans from CartPole to LLM post-training, RLVR, and Agentic RL. Welcome to check it out and share feedback 😆

keyboard_arrow_left Previous keyboard_arrow_right Next

3 33 156 61K 175

𝗧𝗵𝗲 𝗿𝗲𝗰𝗼𝗿𝗱𝗶𝗻𝗴 𝗼𝗳 𝗟𝘂𝗰𝗮𝘀 𝗕𝗲𝘆𝗲𝗿'𝘀 (@giffmana) 𝗹𝗲𝗰𝘁𝘂𝗿𝗲 𝗮𝘁 @ETH 𝗶𝘀 𝗻𝗼𝘄 𝗹𝗶𝘃𝗲 𝗼𝗻 𝗬𝗼𝘂𝗧𝘂𝗯𝗲 𝗳𝗼𝗿 𝗲𝘃𝗲𝗿𝘆𝗼𝗻𝗲 𝘄𝗵𝗼 𝗰𝗼𝘂𝗹𝗱𝗻'𝘁 𝗷𝗼𝗶𝗻 𝘂𝘀 𝗶𝗻 𝗽𝗲𝗿𝘀𝗼𝗻! This past Monday, we had the pleasure of hosting Lucas (@Meta @AIatMeta Superintelligence Labs) for our "Robot Learning: From Fundamentals to Foundation Models" course. He joined us to talk about: "𝗩𝗶𝘀𝗶𝗼𝗻 𝗶𝗻 𝘁𝗵𝗲 𝗔𝗴𝗲 𝗼𝗳 𝗟𝗟𝗠𝘀". Drawing from a remarkable track record in computer vision and multimodal AI (𝗩𝗶𝗧, 𝗦𝗶𝗴𝗟𝗜𝗣, 𝗣𝗮𝗹𝗶𝗚𝗲𝗺𝗺𝗮) 🧠, Lucas delivered a masterclass on the frontier of multimodal foundation model training: from pre-training to post-training, where the field stands today, and what comes next 🚀 📽️ YouTube Recording: youtu.be/0XB7fNS_ONg 📚 Course Website: cvg.ethz.ch/lectures/Robot…

Excited to see my student’s work on Flux Matching out. It turns out you can learn a much broader class of vector fields with the data distribution as stationary (not just the score). This lets you enforce useful properties like fast mixing, and it already works on high-dimensional image datasets!

Peter Pao-Huang @peterpaohuang

3 months ago

Introducing Flux Matching, a generative modeling paradigm that generalizes diffusion models to vector fields that need not be the score function. Enables structural priors in the dynamics, faster sampling, interpretable generation, and more! w/ @StefanoErmon @Xiaojie_Qiu 🧵⤵️

21 155 991 147K 762

Many roughly know how a transformer works To REALLY understand modern neural LMs—MoEs, GPU tiling, kernels, RLHF, data—you need CS336 By @tatsu_hashimoto, @percyliang The 2026 edition appears on yt with ~2 weeks delay youtube.com/playlist?list=… Materials cs336.stanford.edu

keyboard_arrow_left Previous keyboard_arrow_right Next