Chris Maher N7CPM @defilan

@Blackwellboy OK, THIS looks freaking cool. Totally going to play with this over the weekend.

@ashxhart @MiaAI_lab Can’t keep up with it all. So exciting! Can’t get bored in this field hehe

Credit to Local Inference Lab. Their vLLM fork, b12x kernels and CSF repack do the heavy lifting. My part: switchless ring routing in b12x, plus a TP3 loader fix that merged upstream today. github.com/local-inferenc… All served through LLMKube on Kubernetes. Full build guide after part 2.

Moved my DeepSeek V4.1 ring onto the official weights this weekend. Lossless. 2.7× faster at 300K tokens. Spent days building my own 3-bit quant first. Turns out the expert format was never the speed lever. Official MXFP4-CSF on vLLM + b12x, 3 DGX Sparks, switchless ring: • 300K prompt: 345s → 129s • 512K context, needle found at 450K • LiveCodeBench 84 (was 79-83) The catch: decode is 34 tok/s without speculative decoding. Part 2 is freeing enough memory to fit it.

@no_stp_on_snek

@no_stp_on_snek Prost!! Looks like a blast!

Huge thanks to the Gufo team, especially Federico Izzo and Francesco Bozzo, for the engine and for turning around the cache fix so fast. And to Jory Irving for reporting #335 in the first place. If you have a Strix Halo, go star it: github.com/gufo-org/gufo

Follow-up from the Strix Halo lab. The cold-prefill win was only half of it. Long agent sessions with thinking on were re-prefilling the whole conversation every turn. At 200K+ context that meant minutes of waiting for the first token, every single turn. Gufo v0.5.0 fixed it (gufo-org/gufo#335). Same box, same model (Qwen3.8 Flash Next, 262K, MTP), 3 hour soak: time to first token at 180-240K context before: 199 s after: 4.1 s 379 turns in 3 hours vs 102. Zero errors. Restores byte-identical to a cold run. Wrote up the whole thing, roadblocks included, plus a step-by-step build so you can run it on your own Strix Halo. Links below.

@thybead It was! Had it essentially sitting as an overnight option for Foreman in LLMKube but for opencode/agentic coding? Forget about it. I knew there had to be some knobs to turn to make it usable. Love the community out there to help me put this all together. Fun stuff!

The story, in the order things broke: llmkube.com/blog/qwen38-fl… The build, manifests and all: llmkube.com/docs/labs/qwen…

Yeah, it was an 8K prompt: 3.7x faster (22s to 6s to first token). 32K: 4.2x. It grows with depth because Gufo's prefill stays flat (1343 tok/s at 8K, 1184 at 250K) while llama.cpp Vulkan drops from 361 to 101. One note though, single-stream decode is ~7% slower than llama.cpp. The win is prefill, plus 2 agents served at once instead of queued.

Cooking in Defilan Labs 🧪 Same AMD Strix Halo box. Same Qwen3.8 Flash Next. Swapped the engine. 250K-token prompt: 41 min to first token → 3.5 min. That's vs llama.cpp at its best of 4 tuned configs, not defaults. 4-12x faster prefill depending on depth, two agents at once instead of queuing, and it passes our quality gate (needle, battery, LiveCodeBench). Engine is Gufo (MIT). More soon.

Working on DeepSeek V4.1 Flash across my three DGX Sparks at TP3, running on the b12x kernels with a vLLM fork built from source for GB10. Side note: staging 510 GB took 85 min from my NAS and ~12 min per node over the Sparks' ConnectX-7 links. Love it. More when the experiment's done.

@no_stp_on_snek

@MatthewPfab Felt two sizable jolts here in Gig Harbor.

@WhidbeyWXGuy Felt it pretty good here in Gig Harbor

@SeattleWXGuy Felt in Gig Harbor

LocalAI folks see this coming a mile away. OpenAI won’t be the only ones.

Tibo @thsottiaux

a week ago

Hi, Tomorrow we are re-opening the Pro $200 subscriptions to new subscribers, but together with it we are also changing how we calculate the usage for it. In effect, if you do the math, it will net out at half the dollar in API spend compared to the old Pro $200 plan. Now that

7K 2K 22K 17.1M 5K

This is awesome, thanks for running it down! 12/12 byte-identical, plus the 4 concurrent requests matching their serial runs was the check I was most worried about. Wild that the fix took it from 12 to 25.7 on an M2 Max. Nice turnaround from Ash too. Good call on the memory limit note for 32GB. I want to give this a shot on my M4 Max studio which only has 36GB

I spent the weekend putting TensorFold on my M5 Max MacBook and serving it through Kubernetes with LLMKube, the operator that turns NVIDIA, AMD and Apple Silicon into one inference cluster. Qwen3.8-27B went from ~26 tok/s to 227 tok/s, measured from a pod through a normal K8s Service. The output was byte-identical to decoding without drafts. What I didn't expect: • It's not the drafter. llama.cpp with the same DFlash2 drafter tops out at ~59 tok/s. The engine is the win. • Plugging it in took down my daily coding model (llama.cpp 0.5.0 dropped --mlock) and turned up a runtime-selection regression that had been hiding since June. All fixed upstream. • My plan to build custom quants from my own agent traffic didn't survive a control. Q4_K_M and Q5_K_S tied on EvalPlus (81.2% each), and on 3 real LLMKube issues all 9 agent runs passed with tests that bite. TensorFold did it ~3x faster. The engine mattered. The quant didn't. Full weekend log, roadblocks in order, plus a guide if you want this on your Mac 👇 llmkube.com/blog/tensorfol… Huge thanks to @ashxhart for TensorFold.

@thecoldxr Looks like @ashxhart has a fix in place already? Great callout as I only tested on my m5 max for this run.