SGLang @sgl_project

Got to work on this one with the @nvidia team — a genuinely fun systems problem 🙏 EPLB balances MoE experts offline, but the router never emits perfectly balanced traffic, so every live batch still skews load and the EP group waits on its busiest rank. Waterfill + LPLB close that gap at dispatch time, no change to model semantics. ➡️ Waterfill pours the dense shared expert onto lighter ranks instead of every rank paying it locally — near-zero overhead via shared-expert fusion into the DeepEP layout. ➡️ LPLB solves a per-layer LP on-GPU each batch to split redundant-replica traffic optimally and shrink the busiest rank. Throughput up to +7.34%, +4.92% on DeepSeek V4, accuracy fully preserved. Thanks @gazhitt , Fei Liang & Aichen Feng!

LMSYS Org @lmsysorg

3 days ago

🚀 New blog: Improving DeepEP MoE Load Balance in SGLang with Waterfill and LPLB We're introducing two dispatch-time load balancers for DeepEP MoE. Even with EPLB, a single batch still hits ranks unevenly. Waterfill and LPLB fix that residual imbalance at runtime, no change to

4 13 74 11K 39

🎉 NVIDIA just released an NVFP4 checkpoint of GLM-5.2 from @Zai_org, a 744B MoE (40B active) for reasoning & coding. Day-0 support is live in SGLang! 🤝 @nvidia > NVFP4 quantization via NVIDIA Model Optimizer: frontier-class reasoning at a fraction of the memory > Sparse attention with IndexShare indexer for efficient long-context > Ready to serve on Blackwell / Grace Blackwell, run it now with SGLang!

🚀 New blog: Improving DeepEP MoE Load Balance in SGLang with Waterfill and LPLB We're introducing two dispatch-time load balancers for DeepEP MoE. Even with EPLB, a single batch still hits ranks unevenly. Waterfill and LPLB fix that residual imbalance at runtime, no change to model semantics. 1️⃣ Waterfill for the dense shared expert Pours shared-expert work onto lighter ranks (“filling the valleys”) instead of every rank paying it locally. Near-zero overhead via shared-expert fusion into the DeepEP layout. ⚡️ +1.48% to +4.66% on DeepSeek V3/R1 across MMLU, GPQA, GSM8K ⚡️ V4 Flash: 49,253 → 51,677 tok/s (+4.92%) 2️⃣ LPLB for redundant routed-expert replicas EPLB splits hot experts evenly, but live traffic drifts from calibration. LPLB solves a per-layer min-max LP on-GPU each batch to split replica traffic optimally and shrink the busiest rank. ⚡️ +0.84% to +7.34%, strongest when redundant replicas exist (red16/red32) Both preserve accuracy: same logical top-k, identical replica weights, only the physical rank changes. Huge thanks to the @nvidia team for the collaboration!

Come see @adarshxs talks about SGLang on July 25th in Bengaluru!

Aritra @ariG23498

3 days ago

[HF ML Club India] We are proud to announce the second IRL event from the HF ML Club India in Bengaluru. This time we have parterned with the @RedHat_AI India team (particularly their PyTorch Engg team). It is happening on the 25th of July. Talks: > @adithya_s_k speaks on

11 7 84 11K 18

Seems like @lmsysorg / @sgl_project are crushing it as of late! Thanks for this, please don't stop whatever it is y'all are doing <3

LMSYS Org @lmsysorg

4 days ago

🎉 Meet LFM2.5-230M from @liquidai, their smallest model yet at 230M params, but it punches way above its weight. Day 0 Support is live on SGLang! Built on the LFM2 architecture for on-device deployment: > Blazing-fast inference, runs everywhere, from cloud GPUs to low-cost CPUs

0 6 37 5K 4

@lmsysorg SGLang + @krea_ai Krea-2-Turbo = High quality images, local, unlimited, free, in 15 seconds! - Hands, fingers, limbs? check - Legible, correctly spelled, valid text in the image? check - Shade from the sunlight in the correct places? check

Register: luma.com/sglang-orcl

SGLang × Oracle Cloud Infrastructure (OCI): AI Infrastructure Meetup @ ICML 2026 After a busy day at the conference, unwind with fellow engineers, researchers & AI practitioners in Seoul. Lightning talks, demos, food & drinks, and plenty of time to connect. 🎤 Featuring Shi Dong (@ShiDong14, MTS at @radixark), plus speakers from OCI Strategic Customer Engineering. 📍 Near COEX, Seoul 🕠 5:30 - 9 PM Space is limited, register to reserve your spot 👇

A few years ago, the future of artificial intelligence looked dark - proprietary models, proprietary inference services. I joined Modal to fix that. Today, we take a major step forward. Optimized inference, open source, with a click.

Modal @modal

5 days ago

It is not too late to _actually_ own your inference. Introducing: Modal Auto Endpoints.

20 60 445 198K 131

Huge congrats to @modal on Modal Auto Endpoints! 🎉 Owning your inference means owning the code that runs it, and we love seeing that philosophy in action🫡Honored that SGLang powers part of the stack, alongside the work on DFlash speculative decoding 🧡

Modal @modal

5 days ago

It is not too late to _actually_ own your inference. Introducing: Modal Auto Endpoints.

20 60 445 198K 131

🔥DFlash on NVIDIA Blackwell: up to 15x throughput at the same interactivity! Block-diffusion drafting proposes a whole token block in one pass for the target model to verify in parallel, and this is now in SGLang! Migrating from EAGLE is one swap: set spec decode to DFlash + the matching checkpoint. Read the full guide: developer.nvidia.com/blog/boost-inf…

NVIDIA AI @NVIDIAAI

6 days ago

Increase inference performance by up to 15x without sacrificing responsiveness. DFlash, an open source lightweight block diffusion model designed for speculative decoding, delivers up to 15x higher throughput on NVIDIA Blackwell while maintaining the same user interactivity

24 69 624 92K 233

New hardware drops & we gawk over the specs but software is the only way to 5x throughput gains in weeks Coevolution of hardware & software is required to meet token demand without going broke & open source is the only known method to coordinate this level of deep technical work

PyTorch @PyTorch

6 days ago

While SGLang provided Day-0 support for DeepSeek-V4, the collaboration between the @lmsysorg and @NVIDIAAI engineering teams has taken its production performance to the next level. According to the public SemiAnalysis InferenceX dashboard, the GB300 disaggregated lane

8 11 78 31K 31

Day-0 support is the starting point, not the finish line. The SGLang community has always been turning frontier models into fast, production-ready serving systems as quickly as possible. In 2 months, folks achieve 5X throughout speedup in Nvidia GB300. Proud of the @sgl_project and @NVIDIAAI teams pushing this forward.

PyTorch @PyTorch

6 days ago

While SGLang provided Day-0 support for DeepSeek-V4, the collaboration between the @lmsysorg and @NVIDIAAI engineering teams has taken its production performance to the next level. According to the public SemiAnalysis InferenceX dashboard, the GB300 disaggregated lane

8 11 78 31K 31

🚀 New blog: Serving DeepSeek-V4 on GB300 with SGLang: 5x Higher Throughput at the Same Interactivity Since Day-0 Together with @nvidia, we achieved 5X higher throughput at the same interactivity, serving DeepSeek-V4 on GB300 with SGLang. Here's how the DeepSeek-V4 serving frontier moved on the public @SemiAnalysis_ InferenceX dashboard: 1️⃣ 5X throughput on GB300 disaggregated: ~2,200 → ~11,200 tok/s/GPU at ~50 tok/s/user 2️⃣ 2.6X more throughput at 80 tok/s/user with MTP. Curves now hold deep into the high-interactivity range deployments actually target 3️⃣ 2.91X on Blackwell Ultra aggregated at 30 tok/s/user, with 6X+ peak no-MTP throughput 4️⃣ W4A4 MegaMoE: activations now quantized to MXFP4 with negligible accuracy loss 5️⃣ A single FP8-einsum fix lifted MTP acceptance 0.57 → 0.70 Huge thanks to @nvidiaAI @radixark for the deep collaboration on this! SGLang is PyTorch-native, and we're excited to share the full write-up on the @PyTorch blog!

PyTorch @PyTorch

6 days ago

While SGLang provided Day-0 support for DeepSeek-V4, the collaboration between the @lmsysorg and @NVIDIAAI engineering teams has taken its production performance to the next level. According to the public SemiAnalysis InferenceX dashboard, the GB300 disaggregated lane

8 11 78 31K 31

🎉 Meet Krea 2 from @krea_ai, an aesthetic open-source image model ranked #1 text-to-image from an independent lab on @ArtificialAnlys. Day-0 support is now live in SGLang! Krea 2 ships as two models built to work together: 1️⃣ RAW: undistilled base checkpoint: diverse & malleable, made for fine-tuning, post-training & LoRA 2️⃣ Turbo: 8-step distilled checkpoint: fast, high-quality text-to-image Train LoRAs on RAW, run them on Turbo: base for training, Turbo for fast inference on your own hardware Run it now with SGLang!

Krea @krea_ai

6 days ago

today, we release the open weights of Krea 2. welcome Krea 2 Raw and Krea 2 Turbo, an undistilled model from mid-training meant to be fine-tuned, and a fast distilled version with a wide aesthetic diversity. read the details below 👇

88 247 2K 728K 1K

Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs. She also built FlexGen, which made a 175-billion model run on a single consumer GPU, and helped build Chatbot Arena. Three artifacts the whole field uses, one researcher. SGLang hit a 5x cost cut over DeepSeek's own API, and a dozen teams reproduced it. Everyone argues about models. She builds the engines that actually serve them cheaply enough to survive.

h100envy @h100envy

a week ago

x.com/i/article/2067…

3 4 42 49K 63

🎉 Day-0 support for Laguna M.1 from @poolsideai is live in SGLang! This is a 225B MoE built for agentic coding & long-horizon work. 1️⃣ 70-layer MoE: 3 dense SwiGLU layers + 67 sparse MoE layers, 256 experts, top-k=16 with aux-loss-free load balancing 2️⃣ Global attention across all layers: 64 Q-heads, 8 KV-heads, softplus output gating 3️⃣ Native interleaved reasoning: thinking between tool calls, toggleable per-request 4️⃣ Strong on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro & Terminal-Bench 2.0. Run it now with SGLang!

Poolside @poolsideai

2 weeks ago

Today we’re releasing the weights for Laguna M.1, our most capable model to date, with a 256K context length. Both base and post-trained checkpoints are now available on Hugging Face under Apache 2.0.

45 117 1K 624K 582

Try it with SGLang (your preferred inference😁)!

ollama @ollama

2 weeks ago

@Altaf_P7 Try it with @sgl_project or if you have a preferred inference! Cookbooks available: huggingface.co/zai-org/GLM-5.2 We are all working together to make open models viable! Let’s go team open! Ollama’s cloud is my preference but you get choice with openness!

0 0 4 3K 1

MOSS-TTS-Local Transformer v1.5 is now open source 🔥 SGLang-Omni Day0 support @sgl_project @lmsysorg check the blog 👇 mosi.cn/blog/moss-tts-…