Keyu Tian @keyutian

Master's student @ Peking University. Incoming Ph.D. student. linkedin.com/in/keyu-tian/?… Joined October 2021

Tweets

24
Followers

404
Following

21
Likes

39

Keyu Tian @keyutian

3 years ago

@noahcjones here is a brief comparison

0 0 1 3K 0

View Details

Now this paper has been accepted to #ICLR2023 as a spotlight🌟! I truly appreciate everyone's efforts. Check out our repo github.com/keyu-tian/SparK for the latest demo video and updates! Title: "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

Sebastian Raschka @rasbt

4 years ago

How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*! Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ... 1/7

9 101 484 141K 262

4 12 76 36K 32

View Details

Python Trending 🇺🇦 @pythontrending

4 years ago

SparK - [ICLR'23 Spotlight] The first successful BERT-style pretraining on any *convolutional network*; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" github.com/keyu-tian/SparK

0 2 6 10K 4

View Details

/MachineLearning @slashML

4 years ago

: The first BERT-style pretraining on CNNs! #nlp #cnn reddit.com/r/MachineLearn…

1 7 18 11K 9

View Details

Keyu Tian @keyutian

4 years ago

@foozlefoo Thanks, we're glad SparK may be helpful to your project! Sorry we delayed the update for a few days because it's Lunar New Year today, we'll get back to you as soon as they're posted.

1 0 1 944 0

View Details

Keyu Tian @keyutian

4 years ago

PS: our original submission in Oct. 2022 was titled "Sparse and Hierarchical Masked Modeling for Convolutional Representation Learning"; we've changed it to the new one to more clearly express our vision for bringing the powerful BERT-style self-supervised learning to CNNs.

1 0 3 6K 2

View Details

Keyu Tian @keyutian

4 years ago

@rasbt Thank you so much! Your notes are really helpful in letting people know about what we do! And I'm glad to see this work has the opportunity to make a meaningful contribution to our community.

0 0 1 925 0

View Details

Daisuke Okanohara / 岡野原大輔 @hillbig

4 years ago

画像認識の事前学習で、ViTは一部の入力をMaskし、それを予測するタスクが成功していたがConvは隣り合うパッチ間で重複があり情報が漏れ成功してなかった。SparKは出力も入力と同じマスクを維持するSparseConvを使うことで問題を解決、また復号器側も階層を持ちさらに改善 openreview.net/forum?id=NRxyd…

0 9 61 15K 19

View Details

Keyu Tian @keyutian

4 years ago

@reidatcheson @rasbt Thanks for your interest! We are writing a google Colab tutorial to make it easy to play with our pre-trained models (e.g. to reconstruct the images you upload). We'll release it before this weekend and hope it can be helpful.

0 0 2 205 0

View Details

Keyu Tian @keyutian

4 years ago

@rasbt Yeah, that's true for point 2. It'd be better to say "CNNs achieved higher performance than Transformers". BTW, I really appreciate the contrastive idea, which seems more natural for image data than randomly masking. But it turns out the BERT-style pretraining is more effective.

0 0 0 185 0

View Details

Sebastian Raschka @rasbt

4 years ago

9 101 484 141K 262

View Details

fly51fly @fly51fly

4 years ago

[CV] Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling K Tian, Y Jiang, Q Diao, C Lin, L Wang, Z Yuan [Peking University & Bytedance Inc & University of Oxford] (2023) arxiv.org/abs/2301.03580 #MachineLearning #ML #AI #CV [1/2]

1 10 31 9K 18

View Details

Keyu Tian @keyutian

4 years ago

@HappyToKnowThat @jamesr66a it's kinda like the number of neurons in a brain, the more, the smarter. And each parameter is a float in Java, so you can imagine how much memory it takes.

0 0 0 136 0

View Details

Keyu Tian @keyutian

4 years ago

@ducha_aiki Thanks for sharing😃! For those who may be interested in the code implementation, please see: https://github[dot]com/keyu-tian/SparK

0 0 2 115 0

View Details

Keyu Tian @keyutian

4 years ago

@Deep__AI Thanks for sharing😃! For those who may be interested in the code implementation, please see: https://github[dot]com/keyu-tian/SparK

0 0 3 243 0

View Details

DeepAI @DeepAI

4 years ago

🤩The first successful BERT-style #SelfSupervisedLearning on any convolutional network! #ResNet now enjoys masked autoencoding! 🚀A breakthrough paper "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" by @keyutian et al. deepai.org/publication/de…

1 3 8 3K 2

View Details

Dmytro Mishkin 🇺🇦 @ducha_aiki

4 years ago

Designing BERT for convolutional networks: sparse and hierarchical masked modeling Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, Zehuan Yuan tl;dr: create image, which looks to CNN same, as transformers -> MIM starts working arxiv.org/abs/2301.03580…

4 25 116 25K 43

View Details

Keyu Tian @keyutian

4 years ago

@kzmolikova @rasbt Thanks for noticing this, that is our work which's been on OpenReview since Oct. 2022 (https://openreview. net/forum?id=NRxydtWup1S), and uploaded to arxiv recently. It gives some different results from ConvNeXt v2, e.g., our method works pretty well on ConvNeXt v1 and ResNet.