O(1) @JcryptJ

io earth mostly Joined June 2022

Tweets

474
Followers

78
Following

156
Likes

895

O(1) @JcryptJ

3 days ago

"I run my own local AI server. It sucks and will always suck, forever." Depends what you're doing with it. I run Blackwell, everything in vLLM. If you're trying to make your homelab beat frontier on reasoning, you're right, don't bother. But that's not what local is for. Trivia recall isn't an intelligence test. Asking a model about Richard Heart tests long-tail memorization, which is the first thing that gets compressed out of a small model. It says nothing about whether it can follow a schema or pull a phone number out of messy HTML. Any real agent has retrieval attached and isn't answering from weights anyway. Batching isn't an argument against local. vLLM does continuous batching. That's the whole point of it. When my agent fires forty concurrent calls, I get the same utilization curve the providers get. Batching only wins for someone typing one prompt at a time. Right-size to the task, not the leaderboard. Nobody needs half a million in hardware to run the biggest open weights drop. Wrong purchase. The question is the smallest model that clears the bar. A leads agent hammering the Google Places API, normalizing addresses, deduping, scoring fit: low bar, huge volume. Ideal local workload. Break-even is the wrong frame. He's comparing capex against buying the same tokens. Compare it against the runs you never do because the meter makes you flinch. I re-run the whole pipeline five times to tune a prompt and it costs me electricity. That changes what I'm willing to attempt. And some things you can't buy. As an MSP. Client data leaving my network is a compliance conversation I'd rather skip. Local means no egress, no rate limits, and no provider retiring a model and breaking a prompt I spent a week tuning. The real answer is routing. Local eats the ninety percent that's grunt work, frontier handles the ten percent that needs to think. Anyone saying local replaces frontier is wrong. Anyone saying frontier replaces local has never watched an agent loop burn nine million input tokens in an afternoon. 400GB of VRAM sucking isn't a physics result. It's a stack problem.

1 0 3 173 0

View Details

O(1) @JcryptJ

2 weeks ago

@dunik_7 BS, every single token requires re-reading essentially the entire model off disk. A 671B model at 4-bit is 350GB.. per token. Real-world reports put it at 5–30x slower than proper full-VRAM inference and 50–200x slower than APIs seconds-per-token or worse, with CPU pinned at 100%