Junlin Han @han_junlin

Multimodal pre-training and post-training. final-year PhD student at Meta FAIR @AIatMeta and Oxford @OxfordTVG junlinhan.github.io London, England Joined July 2020

Tweets

291
Followers

1K
Following

1K
Likes

2K

Junlin Han @han_junlin

a day ago

@FidlerSanja @VeedaAI @ZGojcic @HuanLing6 Congrats everyone!! 🥳

0 0 1 215 0

View Details

1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free. arxiv.org/abs/2608.08888

16 108 724 260K 694

View Details

Junlin Han @han_junlin

2 weeks ago

@WaikitChan_Baoz thank u Waikit very much!

0 0 0 20 0

View Details

Junlin Han @han_junlin

2 weeks ago

We live in a multimodal world. We see, talk, act, and dream. Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch? Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition So what’s the path forward? Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects/physi…) We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

30 171 1K 106K 715

View Details

Junlin Han @han_junlin

2 weeks ago

@UUUUUsher Thanks so much Quankai!!

0 0 1 46 0

View Details

Junlin Han @han_junlin

2 weeks ago

This is great and super relevant! I really like this hypothesis, it also aligns closely with our findings. Since most vision-language pre-training relies on captioning tasks, which are relatively 'easy,' this mechanism makes a lot of sense. Please let me know once the paper is on arXiv or published elsewhere; we’d love to discuss and highlight it in our work!

0 0 1 39 0

View Details

Junlin Han @han_junlin

2 weeks ago

Thank you Dinkar!! Wow those are very great questions. So far my thoughts: 1. Formation vs persistence: Most representation is learned in pre-training, we do see that models with higher vision activations retain them after SFT (we used the Cambrian-7M dataset). Generally I think keeping a small ratio of vision data during post-training can prevent these circuits from being destroyed. 2. Task-level impact (e.g., shuffled patches): We do see a positive correlation on task accuracy. Models with higher vision activation (like early fusion vs. late fusion under the same vision token budget in Tab 2) perform better on VQA and generation. We haven't tested shuffled patches specifically yet, though! 3. Recovering from laziness via fine-tuning: Probably possible with the right data mix/distribution and very long time training? Though we'd need to avoid regressing language capability...

0 0 0 14 0

View Details

Junlin Han @han_junlin

2 weeks ago

@LiuBenlin 🧊🧊🧊

0 0 0 78 0

View Details

Junlin Han @han_junlin

2 weeks ago

@vhjf36495872 Thank you Jinfa!!!

0 0 1 217 0

View Details

Junlin Han @han_junlin

2 weeks ago

@darakuten117 Thanks a lot Nayuta!!

0 0 1 178 0

View Details

Junlin Han @han_junlin

2 weeks ago

Thanks Xuhui for sharing this work! It looks super interesting! Yeah, aligning the language space (discrete) with the vision space (usually continuous) is very valuable! There are also a couple of works in visual generation that leverage this idea as well. Curious to see how it works when scaled up!

0 0 2 142 0

View Details

Junlin Han @han_junlin

2 weeks ago

@ahlawat_varun Thanks! We highlighted Kimi a couple of times in the paper as representative work in early fusion (first time right at the beginning of the introduction). Sorry X posts don't leave much room for texts so I didn't write any related works. Thanks again for pointing it out!

0 0 1 24 0

View Details

Junlin Han @han_junlin

2 weeks ago

@ubuto23 @_amanda_long Yeah, I think some frontier labs did a lot of exploration on the visual understanding side of unified pretraining (and audio as well). So our value is also on the generation side to provide a more holistic view and a deeper understanding of the underlying mechanisms.

0 0 4 46 0

View Details

Junlin Han @han_junlin

2 weeks ago

@dadadaistt Yea! This is also confirmed in kimi-k3 (for understanding side)! Competition we might still need some arch level or even objective level (diff vs ar) explorations.