-
Tweets105
-
Followers464
-
Following299
-
Likes1K
@finn_fergus yea we have some ideas for this while keeping chain :) mostly tree drafting also complicates the engine implementation especially with overlap scheduling etc
As always, it’s a lot more nuanced than this. In spec decoding, the predominant cost is target verification, the chain vs tree tradeoff is well known, and at production concurrencies you’re often better off training a better draft model than doing tree drafting.
I want to bring your attention to JetSpec because it looks strictly smarter and stronger than previous speculative decoding and block diffusion approaches (yes, again). Avg 1000 t/s single stream with Qwen-8B on B200. Basically, you can better utilize compute at any batch size.
vLLM has even recently removed tree attention support github.com/vllm-project/v…
It is not too late to _actually_ own your inference. Introducing: Modal Auto Endpoints.
@_EldarKurtic @jvmncs @charles_irl we’ve been collaborating with the zlab for a while and this was written before dflash training was well supported in open source
@dysondunbar @charles_irl @Alibaba_Qwen I played too much pokemon champions and forgot. It’s training and will be released later.
@hsu_byron @charles_irl @Alibaba_Qwen I would do it for k2.6 and k2.7 in exchange for a kimi labubu
Speculation Is All You Need. In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x. Over 1k output tps for 3.5 122B-A10B on a B200. Read the blog for why we're all-in on spec dec. modal.com/blog/spec-is-a…
Try them out! 🚀 Really enjoyed the amazing collaboration with the Modal team. The retrained DFlash draft models now support longer contexts, making them a better fit for agentic workloads.
Speculation Is All You Need. In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x. Over 1k output tps for 3.5 122B-A10B on a B200. Read the blog for why we're all-in on spec dec. modal.com/blog/spec-is-a…
🚀 New blog: The next generation of speculative decoding: DFlash and Spec V2 DFlash + Spec V2 hit >4.3X baseline throughput for LLM inference, now the default speculative decoding engine in SGLang! Together with @modal and z-lab.ai, our jointly-released DFlash drafter for Qwen 3.5 397B-A17B beats both baseline and native MTP in every setting we benchmarked: 1️⃣ >4.3X baseline & 1.5X native MTP throughput (concurrency 1, HumanEval, 8xB200) 2️⃣ Block diffusion drafter: a full token block in one forward pass 3️⃣ KV injection: target-model features fed into every draft layer’s KV cache for higher acceptance 4️⃣ Spec V2 overlap scheduler: +33% end-to-end Read the code, deploy a DFlash server, and start experimenting!
🚀 DFlash now runs on SGLang's new default speculative-decoding engine, Spec V2. ⚡️ Hitting >4.3× baseline throughput (1.5× over native MTP) on Qwen 3.5 397B-A17B. Same quality, more speed! ⭐ github.com/z-lab/dflash
couldn't have asked for better people to do this with. @sgl_project @modal and the z-lab team @jianchen1799 @yesheng_liang @zhijianliu_ 💚
9+ accept lengths on coding workloads generic drafter btw qwen 397b 4x faster repro btw dflash go brrr
We worked with @lmsysorg and z-lab.ai to - integrate DFlash spec into @sgl_project - make it faster with overlap - train a DFlash drafter for @Alibaba_Qwen 397B-A17B The result: up to 4.3x greater throughput over baseline and 1.5x over native MTP.
Great to see DFlash is being used in MiMo!
🚀 1,000+ TOKENS/S ON A 1T MODEL! 🚀 We are thrilled to release Xiaomi MiMo-V2.5-Pro-UltraSpeed in collaboration with @TileRT_AI , breaking the 1,000 tokens/s output speed on a 1 Trillion parameter model for the FIRST TIME! Not wafer-scale integration like Cerebras. Not pure
Amazing to see DFlash helping push Xiaomi MiMo to 1,000+ tokens/s on a 1T model. Happy to see our work supercharging one of the best open-source LLMs! 🚀
🚀 1,000+ TOKENS/S ON A 1T MODEL! 🚀 We are thrilled to release Xiaomi MiMo-V2.5-Pro-UltraSpeed in collaboration with @TileRT_AI , breaking the 1,000 tokens/s output speed on a 1 Trillion parameter model for the FIRST TIME! Not wafer-scale integration like Cerebras. Not pure
me with 0 commits to the modal monorepo 👀
@schrockn Yeah agree PR count is slightly better metric than LoC
📗 @__the__human__
28 Followers 4K Following
Josh Harkins-Finn @JHarkFinn
2 Followers 7K Following
Shrinav Loka @tech_cultist
85 Followers 355 Following Prev @Stripe | CS @ UMD | Grinding Comp Pro + Hackathons + PyTorch | GPUs go brrrrrr
athena77 @athenaseven7
8 Followers 47 Following
brandon galang ▲ @brandon_galang
4K Followers 2K Following ▲ applied ai, gtm engineering @vercel • ex @amazon @intel • mba @yalesom opinions my own
Kasra @kasra_danesh
450 Followers 832 Following 🌁 Building @sfcompute, Prev led 4 products at @Rocketberlin
Zavain Dar @zavaindar
13K Followers 3K Following Batman; Investor X Computer Scientist. Founder @_DimensionCap
Phantom Axion 🥷 @PhantomAxion
909 Followers 204 Following 🥷 PulseChain privacy hunter zk · nullifiers · Kohaku · ProveX · Railgun · RH privacy • Day 1 PulseChain OG→ now untraceable only. Prove it, then vanish.
johnny, skroderider e... @generativist
19K Followers 9K Following Lapsed computational social scientist. Blasphemous orthodox jester with Discordian allegiances, nerdy habits, burrito affinities, and a big computer. 🇺🇸
M @martellmalone
113 Followers 183 Following
Dan Saunders @djsaunde
883 Followers 3K Following research eng / perf at @LumaLabsAI prev @UnslothAI @axolotl_ai @awscloud @UMassAmherst small homestead owner 🐑🐓
Carsten Lindstedt ஃ @clindstedt
651 Followers 5K Following AI Innovation Enablement at Dassault Systèmes; ex Creative Director at Razorfish / startup co-founder (acq. by Bosch), 25 yrs of experience
Ben Geist @b_geist
722 Followers 426 Following Research Eng @ramplabs / physics + math nerd / Kate Bush fan
omgbtc @OMGBTC
751 Followers 694 Following I automate financial networks and untangle slinkies. #bitcoin obviously
ST @samx3499
19 Followers 1K Following
Vaclav Cerny @vacla_vcerny
10 Followers 1K Following
Deepesh Bansal @deepeshbansal_
21 Followers 234 Following AI@ https://t.co/iLcsE0dMPK | BITS Pilani | tinkering in GenAI
🍼 @C3ddan
3 Followers 835 Following
TO @Oczkos
116 Followers 2K Following
Matt Porter @mabbikeel
49 Followers 466 Following Software developer. Interests: Python, new languages, database tech
Hari Subbaraj @hsubbaraj
95 Followers 156 Following
Product Gremlin @productgremlin
38K Followers 1K Following CEO & Head of Product @gremlinlabs "Vive ut Gremlinus mischivus; morere ut Gremlinus magnificus" Powered my mischief. 💚 #vibecoding #entrepreneur
iCjay𝕏 @iCjayX
6 Followers 372 Following Fortified 8 essential nutrients and vitamins. 99.9% pure unfiltered.
Alex Cheema @alexocheema
49K Followers 3K Following building @exolabs | prev @UniOfOxford We're hiring: https://t.co/UlkApFndnH
Daniel @dansumptr
0 Followers 2K Following
Fdjheh @Fhdjheh
3 Followers 321 Following
Cadence Agyirey 🇬�... @kubedoll
514 Followers 759 Following Founder @ https://t.co/Br2DfsprN2 | Serving open source models & building the future of inference
Prahaladh Chandrahasa... @pralichan
40 Followers 480 Following Founding Engineer making AI Agents |Prev adventures @scsatcmu| ML Systems| AI Safety
Victor @space3709
17 Followers 83 Following i like gpus and making software (and occasionally hardware) go brr. 20 - CS & Math @ UMass Amherst
Claire Chen @ritzyshark31
0 Followers 37 Following
taesiri @taesiri
980 Followers 6K Following Research Scientist @ EA Sports, VLMs, Evals, All opinions are my own.
anonymous @An0niim0u5
14 Followers 1K Following
Yiran Wang @nlp_yiran
97 Followers 2K Following Natural Language Hasher (0xDC687816 | 0x54D | 0xCA29)
VegetaAvatar @VeGeTaX29
20 Followers 7K Following
Mingyi Lu @mingyilu123
327 Followers 134 Following Head of Product @radixark @lmsysorg @sgl_project | ex-AI researcher @Stanford, @TuSimpleAI, PM @tiktok_us | 🔮 Building AI infra that scales
Hari Subbaraj @hsubbaraj
95 Followers 156 Following
Banghua Zhu @BanghuaZ
8K Followers 1K Following Cofounder & CTO @radixark. Assistant Professor @UW. Prior @Nvidia @Berkeley_EECS
Baizhou Zhang @baizhou_zh83925
241 Followers 107 Following Working at RadixArk, SGLang Core Contributor | Ex-Intern at Nvidia, Baidu Linkedin: https://t.co/zWu1LAMuLR
Mingyi Lu @mingyilu123
327 Followers 134 Following Head of Product @radixark @lmsysorg @sgl_project | ex-AI researcher @Stanford, @TuSimpleAI, PM @tiktok_us | 🔮 Building AI infra that scales
Xiaomi MiMo @XiaomiMiMo
29K Followers 40 Following @XiaomiMiMoDevs where we share API and product updates
Eric Hansander @erichansander
263 Followers 977 Following Building things @modal. ex-founder @twirldata
Quanquan Gu @QuanquanGu
26K Followers 2K Following Professor @UCLA, Ex-ByteDance Seed | Recent work: Seed2.0, SeedFold, SeedProteo | Opinions are my own
Crystal @crystalsssup
26K Followers 1K Following @Kimi_Moonshot Staff. UCSD Alum. Personality goes a long way. Be useful.
Ryan Fleury @rfleury
24K Followers 130 Following RJF // @dgtlgrove // Making RAD Debugger at @radgametools in @epicgames (opinions my own)
Dan Fu @realDanFu
8K Followers 244 Following VP, Kernels @togethercompute Assistant Professor @ucsd_cse Looking for talented kernel engineers and performance engineers!
Andreas Abel @uops_info
704 Followers 46 Following
Saining Xie @sainingxie
40K Followers 2K Following cofounder & chief science officer at @amilabs | faculty @nyu_courant | prev: @googledeepmind @meta (fair) @ucsandiego | ynwa
Yu Zhang 🐙🌘 @yzhang_cs
5K Followers 872 Following @Kimi_Moonshot; working on efficient methods for LLMs; disciple of parallel programming; INTP; 🐈
SAIR @SAIRfoundation
4K Followers 35 Following Terence Tao & Nobel, Turing, Fields laureates advancing scientific discovery & guiding AI with scientific principles. Grounding intelligence. Scaling discovery.
Shankha @ShankmanJunior
53 Followers 358 Following
Stefano Ermon @StefanoErmon
142K Followers 357 Following AI Prof @Stanford | CEO & Cofounder @_inception_ai | Co-inventor of DDIM, FlashAttention, DPO, GAIL, and score-based/diffusion models
Michael Waskom @michaelwaskom
6K Followers 138 Following Machine pedagogue and data aesthete. Creator of https://t.co/imXawO2glS. Currently @modal, building software for AI.
Avner May @avnermay
1K Followers 249 Following Staff Research Scientist at https://t.co/WEMkSSRVeZ. Formerly research scientist at Google, postdoc at Stanford, and PhD student at Columbia.
Tanishq Kumar @tanishqkumar07
5K Followers 81 Following CS PhD student @StanfordAILab. prev math undergrad @Harvard.
Kaixin Li @kxli_2000
6K Followers 153 Following Final year PhD @ NUS Prev Qwen, Tiktok, Sea AI Lab Agents, Code and Multimodality Core contributor of Qwen3.5 & VL &Coder 🖥️ScreenSpot-Pro
Mark Chen @markchen90
74K Followers 353 Following Chief Research Officer at @OpenAI. Coach for the USA IOI Team.
Alec Radford @AlecRad
71K Followers 303 Following
Patrick C Toulme @PatrickToulme
8K Followers 281 Following TPU/XLA @google. Formerly worked on Meta MTIA and AWS Trainium. Opinions are my own. All blogs shared are personal work.
Ke Li 🍁 @KL_Div
6K Followers 423 Following Assistant Professor of Computing Science @SFU. Ph.D. from @Berkeley_EECS and Bachelor's from @UofTCompSci. Formerly @GoogleAI and Member of @the_IAS.
Rui Ueyama @rui314
56K Followers 342 Following 米国Googleでソフトウェアエンジニア ＆ スタンフォード大学コンピュータサイエンス修士 → 自分の会社 → 東京大学コンピュータ科学博士課程（現在） Creator of LLVM lld and the mold linkers
xjdr @_xjdr
29K Followers 689 Following building AI that wont embarrass me in front of my own standardsTrends for United States
You might like
