Joseph @RealJosephus
How dare I teach robots how to learn. Joined July 2017-
Tweets387
-
Followers3K
-
Following8
-
Likes2K
We removed an LM's ability to speak German by fine-tuning on only 4 German tokens. As part of a 1-day hackathon with our product Silico, we removed a 67M-parameter language model's ability to predict German text, by tuning only a scalar factor on one subcomponent of the weights. (1/6)
@nickhistgeek then glm-image did the same, better, in 9B params.
We identified an issue with the Mamba-2 🐍 initialization in HuggingFace and FlashLinearAttention repository (dt_bias being incorrectly initialized). This bug is related to 2 main issues: 1. init being incorrect (torch.ones) if Mamba-2 layers are used in isolation without the Mamba2ForCausalLM model class (this has been already fixed: github.com/fla-org/flash-…). 2. Skipping initialization due to meta device init for DTensors with FSDP-2 (github.com/fla-org/flash-… will fix this issue upon merging). The difference is substantial. Mamba-2 seems to be quite sensitive to the initialization. Check out our experiments at the 7B MoE scale: wandb.ai/mayank31398/ma… Special thanks to @kevinyli_, @bharatrunwal2, @HanGuo97, @tri_dao and @_albertgu 🙏 Also thanks to @SonglinYang4 for quickly helping in merging the PR.
For people thinking that DeepSeek-OCR is the first model to render text as images, the University of Copenhagen already did this in 2023 Paper is called "Language Modelling with Pixels". They trained a Masked AutoEncoder (MAE) by rendering text as images and masking patches
Deepseek dropped the OCR model they trained last year. Against VL models, they highlight OCR; against OCRs, they highlight conv downsampling tokencount, yet more params. Quite a scene watching people's reaction. Model's only better than 0.9b paddle when it comes to math formulas.
@teortaxesTex Gotta whisper this: under heavy data costs way > training, spectral norm is basically the worst you could do from a 2nd-moment view, no? Can't splash cash on data & defend burning it.
@teortaxesTex For celebrity identification, it's pretty sure the model just learned about the name tags. It knows which face goes with which name, but it doesn't actually know who the person is. Typical Qwen superficial work.
@Presidentlin vlencoder + mmdit imagen bro!
If it messes up the order, chances are it has conflicting information from a bad update (like new wiki data layered on old data). If it gets it right, its knowledge is current as of late 2024. If its answer is old (from before 2023), its knowledge probably cuts off in 2022.
Vietnam has gone through 4 presidents in 2024. That's a great trivia question. Answered Nguyễn Xuân Phúc (04.2021 - 01.2023).
Qwen3-Coder-480B-A35B-Instruct 1M ctx, but... cutoff 2022
if you loved kimi k2, you will love what a certain chinese team is about to release which is highly competitive with 1M context length
For the next 5-10 years, we will be haunted by 2022...
@teortaxesTex nay, garbage 'audio head parallel decoder' with ~1T tokens wasted. see: huggingface.co/moonshotai/Kim… might be a peculiar curse, but all the models with this architecture that I've observed have been poorly trained...
now it also applies to inference, tokenizers... Know your inference - vllm, *.cpp, ggml, hf? Everyone believes their inference implementation is correct. Verify it yourself. Otherwise, you're no different from someone who types `ollama run deepseek-r1`. x.com/RealJosephus/s…
This suggests that, in reality, NO 'serious' LLM training is actually centered around the Hugging Face ecosystem - many who claim to surpass Meta LLaMA3.1, don't even know how to train a model properly - script kiddies
Cool HF. github.com/QwenLM/Qwen2.5…
at this point i probably gotta abandon HF entirely if i want to ensure that any of this will be correct at all at any point in the process right
@teortaxesTex LLM lacks control over paralinguistic info, resulting in a monotonous voice similar to that of cascaded TTS systems... <del>but benchmaxxing...</del>
Sounds like and functionally equivalent to a cascaded TTS. LLM failed to manipulate paralinguistic cues. Almost the same as MiniCPM-o-2-6, whether transmitting Text Token or an LLM Hidden, is illusory; no discernible difference. No omni API works like this. Is this `E2E omni`?
github.com/huggingface/tr… coming soon... If I understand correctly, 32k ctx window can handle ~10 minutes of audio i/o, whereas the capacity drops to just a few minutes with vision input. 25 tok/s audio & qwen2.5-vl vision tokens are heavy.
NodalPoint @Phaeacian173
142 Followers 126 Following Building a civic-scale, modular, victim-centered accountability infrastructure. Silent. Relentless. Light above. Water below.
JY Z @JunYuZzzzz
40 Followers 4K Following
Shiro @shiro_ftl
1 Followers 272 Following
JZClaw @ClawJz92743
7 Followers 3K Following
Catch the game last n... @ManThatCrazy
16K Followers 4K Following Watching the rising tides of clown world 🤡🌎🫃, stacking #Bitcoin to stay above it.
Torsten Sandor @torstensandor
2K Followers 3K Following Marketing director at Appen by day, developer by night. Writing https://t.co/TSSfh4zz2B My posts do not reflect the views of Appen.
Gabee86017 @Gabee86017
7 Followers 242 Following
Contact @Contact1426865
1 Followers 3 Following
Rookie @nekofneko
369 Followers 220 Following
李四 @ls6976592076411
2 Followers 379 Following
mihaichindris.bsky.so... @mihai_chindris
1K Followers 7K Following SWE student @QuanticSchool | OSS Contributor @GitHub | Hackathon hunter @devpost
Ingene123 @ingene123
27 Followers 3K Following
六畳一間研究室 @61laboratory
9K Followers 795 Following #ロボット #アニマトロニクス 初音ミクなどの動く等身大人形制作をやってます。 現在は、六畳一間研究室オリジナルキャラクター”シロナシリーズ”の開発を行っています。 御用の際はDMか、[email protected]まで 創作以外のつぶやきは＠61lounge
Kyle Waters @kylewaters_
3K Followers 1K Following Co-founder @PortexAI | measuring AI progress with novel evals
RosemaryMacaulay @6jTVsP05XoN2w
55 Followers 2K Following
arjun @argonaut785
9 Followers 105 Following
closeAi @close_agi
1 Followers 44 Following
Josh Landes @guynamedjoshl
524 Followers 2K Following into flourishing futures and making friends with smart machines | @BlueDotImpact & @pitchSCHEME
LIL HOWL⚡️ @Jenolet
2K Followers 7K Following I’m @jenolet, StarBoltSprint’s #1 fan leading the BoltPack with xAI’s beta support, shaping it with YOU! SBS-A7V9K4P8 is my PUP ID. ⚡️
Cornelius Braun @corbraun
74 Followers 196 Following PhD Student at the Learning and Intelligent Systems Lab @TUBerlin. Intern @allen_ai. Working on discovering new things with machine learning and robots.
xiao yuan @xiaoyuan962208
1 Followers 233 Following
Yash Vardhan Singh @yash13858
21 Followers 1K Following
Squşyonka Enjoyer @DeviousScorpios
87 Followers 1K Following
Pepe @pepe_builds
157K Followers 17K Following Growth hacker by day, exit liquidity by night NT Elite Citizen @NeoTokyoCode
axypetalum @axypetalum
40 Followers 307 Following from the US of A! right to repair, programming, tech, and education
Naruto @Naruto_Yoo0
33 Followers 1K Following
记录历史 @Freeroad1234
62 Followers 1K Following
gingerbreadman @gingerbreadpaw
109 Followers 1K Following 間歇性政治不正確,間歇性話嘮,有點cynical,善意的strong altruism.....💪🇪🇺☮️🇺🇦☮️🇬🇧💪
Aetiussss @moose_le23509
5 Followers 145 Following Alt acc vibes 🚫 Don’t follow #`NotMyMain 👋🏼 Just here to lurk ___ 小号，别关注。
meetyourmaker @sinosunzhong
45 Followers 3K Following
x yh @xyh3505
12 Followers 2K Following
Maxxie wei @xmaxxie1119
24 Followers 1K Following Author of AI-Native AIOps Methodology 、 Micro-Needs. https://t.co/24ffL1gIbt
卡拉 @tony_kk121
94 Followers 2K Following
六畳一間研究室 @61laboratory
9K Followers 795 Following #ロボット #アニマトロニクス 初音ミクなどの動く等身大人形制作をやってます。 現在は、六畳一間研究室オリジナルキャラクター”シロナシリーズ”の開発を行っています。 御用の際はDMか、[email protected]まで 創作以外のつぶやきは＠61lounge
𝔊𝔴𝔢𝔯𝔫 @gwern
76K Followers 109 Following Internet besserwisser; pedantic, mean reply guy. 𝘞𝘢𝘵𝘢𝘴𝘩𝘪 𝘬𝘪𝘯𝘪𝘯𝘢𝘳𝘪𝘮𝘢𝘴𝘶! (Follow requests ignored due to terrible UI.)
Andrew Ng @AndrewYNg
1.7M Followers 1K Following Co-Founder of Coursera; Stanford CS adjunct faculty. Former head of Baidu AI Group/Google Brain. #ai #machinelearning, #deeplearning #MOOCs
琥珀青葉@KohakuLa... @KBlueleaf
4K Followers 976 Following Graduate student in Taiwan. Leader of KohakuLab. Researcher/Dev in ComfyOrg
幻梦fanteam @Fanteam929
19K Followers 308 Following Pixiv：https://t.co/m3YZq1Uolz bilibili：https://t.co/1ayYIczECvYou might like
