Jeremy Cohen @deepcohen

Research fellow at Flatiron Institute, working on understanding optimization in deep learning. Previously: PhD in machine learning at Carnegie Mellon. jmcohen.github.io New York, NY Joined September 2011

Tweets

1K
Followers

6K
Following

998
Likes

1K

Jeremy Cohen @deepcohen

9 hours ago

@valeriechen_ @khodakmoments Congrats!!

0 0 0 168 0

View Details

Jeremy Cohen @deepcohen

a week ago

@jonasgeiping I can't tell if this is a play on Houdini or Carlini

0 0 4 247 1

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@Shiwei_Liu66 Would be interested in hearing others’ wild speculation

0 0 0 615 0

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@Shiwei_Liu66 Update: not the secret sauce (definitely known to the other labs)

Shiwei Liu @Shiwei_Liu66

2 weeks ago

@deepcohen Thanks for your kind thoughts and support, Jeremy! I do believe this direction could potentially open up a new level of scaling toward deeper models with stronger reasoning capabilities. To be clear, I was not aware of the Depth-MP work when developing this paper (shamed); the

0 2 12 2K 16

1 0 6 1K 1

View Details

Jeremy Cohen @deepcohen

2 weeks ago

Did Anthropic get more gains out of model scaling than other labs thought was possible? It reminds me of an interesting recent paper, which showed that deep layers in open LLMs are not doing much, and that this can be fixed by scaling the LayerNorm output. arxiv.org/abs/2502.05795

9 22 297 25K 351

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@mrtnm If someone forced me to bullshit an explanation with a gun to my head, I would say that implicit curvature regularization promotes "feature learning" (whatever that is), and that in the fine-tuning setting, you don't want feature learning.

1 0 0 54 0

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@mrtnm Sorry, I missed this. Actually, even when you fine-tune to the same SFT loss, lower LR's are better for fine-tuning: arxiv.org/abs/2604.13627. This indicates that it's not about distance traveled, but rather about implicit bias.

1 0 0 52 0

View Details

Jeremy Cohen @deepcohen

2 weeks ago

The recent Microsoft AI report noted that too much learning rate decay during pretraining hurts post-RL performance. This is actually just the latest of several papers this year pointing out that small learning rates can be harmful in LLM pretraining. (Thread)

6 23 193 18K 133

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@DimitrisPapail rest can be SGD

0 0 1 64 0

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@DimitrisPapail they say that you only need adam for the last layer and the layernorm parameters

1 0 1 76 1

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@SunnySanyal9 The first paper I read with a sqrt(L) or sqrt(l) scaling was this one: arxiv.org/abs/2010.12859. But the paper I tweeted is proposing something different from the others.

1 0 7 616 3

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@brunorganised @DeyNolan It's kind of similar, but these small things can make huge differences, no? x.com/deepcohen/stat…

Jeremy Cohen @deepcohen

2 weeks ago

@xidulu I mean, it sounds similar, but there could be a big difference. Also, this one is layer index rather than depth.

1 0 1 865 0

1 0 1 748 0

View Details

Jeremy Cohen @deepcohen

2 weeks ago

@xidulu I mean, it sounds similar, but there could be a big difference. Also, this one is layer index rather than depth.

1 0 1 865 0

View Details

Jeremy Cohen @deepcohen

2 weeks ago

Oh also, in the *multi-epoch* LLM setting, there is evidence that larger LR's yield better population pretraining loss, exactly mirroring what was known in 'classical' image settings (arxiv.org/abs/2306.08590). (This experiment is with batch size, but LR should be the same)

0 1 5 544 1

View Details

Jeremy Cohen @deepcohen

2 weeks ago

Nevertheless, hyperparameters matter, and I'm glad we're starting to see good science about LR schedules for LLMs. PS: as noted by Catalan-Tatjer et al, weight-averaging recovers many benefits of LR decay but without increasing sharpness. More should consider weight-averaging!