Fred @probablyspiking

@_arohan_ @zacharynado One practical/implementation advantage is that shampoo works in grad space and that’s exposed by every library without fiddling. Is this why shampoo gets played with so much more than KFAC or is it also better? Would love @_arohan_ s take :)

2 0 3 668 2

View Details

Fred @probablyspiking

a month ago

@natolambert Largely agreed. Except they are kind of special 😂

0 0 0 198 0

View Details

Fred @probablyspiking

2 months ago

@gowthami_s Most likely it’s not. But it is all u need

0 0 0 24 0

View Details

Fred @probablyspiking

2 months ago

@giffmana @pengzhangzhi1 Wait, the learning rate matters?

0 0 2 14 0

View Details

Fred @probablyspiking

3 months ago

Better than going for a long walk

0 0 0 16 0

View Details

Fred @probablyspiking

3 months ago

Why does making slop feel so good, if god didn’t mean us to

1 0 0 18 0

View Details

Fred @probablyspiking

7 months ago

@BlackHC Does Adam also work worse with a small lr?

0 0 0 18 0

View Details

Oriol Vinyals @OriolVinyalsML

2 years ago

For Reinforcement Learning to be successful on top of LLMs, it is critical to have a very powerful and accurate reward model. Reward in most language tasks isn't as clearly defined as, say, in chess, where the winning condition is a simple computation. Generative reward models are the most powerful class as they are themselves general. It's great to see how the best LLMs are starting to top this important benchmark! huggingface.co/spaces/allenai…

4 31 197 31K 108

View Details

Jeff Dean @JeffDean

2 years ago

Gemini 1.5-Pro is a pretty good reward model (tops among generative models and second overall in the Reward Bench Leaderboard).

Aliaksei Severyn @aseveryn

2 years ago

Gemini 1.5 Pro when zero-shot prompted to perform an LLM-as-a-judge task ranks 1st when compared to other Generative RMs and 2nd best overall vs other dedicated RMs: huggingface.co/spaces/allenai… (make sure to click on the Generative checkbox).

0 11 62 75K 24

15 34 264 71K 62

View Details

Fred @probablyspiking

2 years ago

@y0b1byte creating fractals

0 0 2 40 0

View Details

Fred @probablyspiking

3 years ago

@ultrono519 Would add a drizzle of lime and nachos before crushing

0 0 1 116 0

View Details

yobibyte @y0b1byte

3 years ago

Are you an AI researcher who's having a baby? Here are some nice names for your child: Transformer Juergen HAL SoftMax Jax Skynet Juergen Juergen Good luck. I hope they don't drop out from school later on.

Jay Cummings @LongFormMath

3 years ago

Are you a mathematician who's having a baby? Here are some nice mathematical names for your child: Epsilon Abscissa Pollygon Quintic Abacus Lemma Vector Max Eureka Tessellate Jacobian Eigen Seven 🧮

61 52 473 348K 47

7 1 27 8K 3

View Details

Fred @probablyspiking

3 years ago

@MSiggel story=paper?

0 0 0 47 0

View Details

Johannes Oswald @oswaldjoh

4 years ago

Transformers learn based on data given in-context! But how? In a new 📜 (arxiv.org/pdf/2212.07677…), we show how to construct self-attention weights to implement gradient descent learning! But does optimization actually find these weights? Surprisingly, yes! A Thread