Claude: Hey, mind if I grep -ohP "useEffect\(.*?\[\K[^\]]+" **/*.tsx 2>&1|tr ',' '\n'|awk 'NF{$1=$1;a[$0]++}END{for(k in a)print a[k],k}'|sort -rn|head -20
Me: ... yeah go for it dude
I don't have too too much to add on top of this earlier post on V3 and I think it applies to R1 too (which is the more recent, thinking equivalent).
I will say that Deep Learning has a legendary ravenous appetite for compute, like no other algorithm that has ever been developed in AI. You may not always be utilizing it fully but I would never bet against compute as the upper bound for achievable intelligence in the long run. Not just for an individual final training run, but also for the entire innovation / experimentation engine that silently underlies all the algorithmic innovations.
Data has historically been seen as a separate category from compute, but even data is downstream of compute to a large extent - you can spend compute to create data. Tons of it. You've heard this called synthetic data generation, but less obviously, there is a very deep connection (equivalence even) between "synthetic data generation" and "reinforcement learning". In the trial-and-error learning process in RL, the "trial" is model generating (synthetic) data, which it then learns from based on the "error" (/reward). Conversely, when you generate synthetic data and then rank or filter it in any way, your filter is straight up equivalent to a 0-1 advantage function - congrats you're doing crappy RL.
Last thought. Not sure if this is obvious. There are two major types of learning, in both children and in deep learning. There is 1) imitation learning (watch and repeat, i.e. pretraining, supervised finetuning), and 2) trial-and-error learning (reinforcement learning). My favorite simple example is AlphaGo - 1) is learning by imitating expert players, 2) is reinforcement learning to win the game. Almost every single shocking result of deep learning, and the source of all *magic* is always 2. 2 is significantly significantly more powerful. 2 is what surprises you. 2 is when the paddle learns to hit the ball behind the blocks in Breakout. 2 is when AlphaGo beats even Lee Sedol. And 2 is the "aha moment" when the DeepSeek (or o1 etc.) discovers that it works well to re-evaluate your assumptions, backtrack, try something else, etc. It's the solving strategies you see this model use in its chain of thought. It's how it goes back and forth thinking to itself. These thoughts are *emergent* (!!!) and this is actually seriously incredible, impressive and new (as in publicly available and documented etc.). The model could never learn this with 1 (by imitation), because the cognition of the model and the cognition of the human labeler is different. The human would never know to correctly annotate these kinds of solving strategies and what they should even look like. They have to be discovered during reinforcement learning as empirically and statistically useful towards a final outcome.
(Last last thought/reference this time for real is that RL is powerful but RLHF is not. RLHF is not RL. I have a separate rant on that in an earlier tweet
x.com/karpathy/statu…)
DeepSeek (Chinese AI co) making it look easy today with an open weights release of a frontier-grade LLM trained on a joke of a budget (2048 GPUs for 2 months, $6M).
For reference, this level of capability is supposed to require clusters of closer to 16K GPUs, the ones being
@thisisdoozy It takes about a good 3-4 days to get back in the flow. When I first came back this earlier this week I was going through it lol. You’ll be good!
.@ilyasut full talk at neurips 2024 "pre-training as we know it will end" and what comes next is superintelligence: agentic, reasons, understands and is self aware
5K Followers 4K FollowingRadiologist, MD. Responding to posts about complexity, memory, time, consciousness, cancer, and Alzheimer’s—
Quantum biology, medical phylosophy.
767 Followers 75 FollowingApplied AI product lab. Investor. Studying math daily. Ex: Dir. Product @GoDaddy, misc startups, Captain @USMC. Dad of 2 & married to @corrie_mays 18yrs ❤️
11 Followers 147 FollowingNormal humans don't interest me. If anyone here is an alien, a time traveler, slider, or an esper, then come find me! That is all.
23 Followers 238 FollowingWelcome to _Shop with a Touch! 🌟 Here, we offer a unique collection of custom-designed products at low prices on Amazon, all while maintaining high quality!
182 Followers 1K Followingcomputers, math, fitness.
i want knowledge and big bags, then going on adventures (science, exploration, biz, sport, whatever)
259 Followers 355 Followingold 🇨🇦 dude trying to find something new. Former-Ex Software consultant, grinding ML, math.
to be clear I drink coffee and code, I don't code android UIs
590K Followers 52K FollowingSan Francisco/Silicon Valley AI | Robots, holodecks, BCIs, analysis of new things | Ex-Microsoft, Rackspace, Fast Company | Wrote eight books about the future.
54K Followers 1K FollowingArmin's handler at https://t.co/B05ybKGkzx. Old man yelling at Claudes.
https://t.co/Q1wG57v1yc
https://t.co/mnOoWUr0TO
https://t.co/8i5vIRE0Wn
1.6M Followers 2 FollowingClaude is an AI assistant built by @anthropicai to be safe, accurate, and secure. Talk to Claude on https://t.co/ZhTwG8d1e5 or download the app.
41K Followers 16 FollowingI've been in the industry for O(40) years and have written O(1M) LOC. I don't think I'll ever write O(another) line again, but I'll be launching more than ever.
1.5M Followers 148 FollowingValue investor | 10+ years of finding undervalued stocks | Founder & CEO @InTheAssembly (the #1 private finance community in the world)
1.2M Followers 787 FollowingProfessor at NYU & Executive Chairman at AMI Labs.
Ex-Chief AI Scientist at Meta.
Researcher in AI, Machine Learning, Robotics, etc.
ACM Turing Award Laureate.
29K Followers 7K FollowingI exist, but at what cost? privacy nerd, maker of cursed art, shitposter, hacking the planet for chaotic good. all nodes are equal.
447K Followers 6K FollowingChief Scientist, Google DeepMind & Google Research. Gemini Lead. Opinions stated here are my own, not those of Google. TensorFlow, MapReduce, Bigtable, ...
16K Followers 0 FollowingTips and tricks for Burp Suite Pro
Managed by @Agarri_FR | Not affiliated with @Portswigger
More free resources at https://t.co/MWqXmV66lr
28K Followers 628 FollowingWeb hacker and Burp Suite Pro trainer
Refer to https://t.co/D5tRH7U2hg for trainings
Follow @MasteringBurp for free tips and tricks