BURKOV @burkov

Books: https://t.co/0EmPM3De9B & https://t.co/45NGbbXIzC App: https://t.co/n2jvMtYhVm PhD in AI, author of 📖 The Hundred-Page LMs Book & The Hundred-Page ML Book linktr.ee/burkov Québec, Canada Joined June 2009

Tweets

24K
Followers

58K
Following

121
Likes

11K

BURKOV @burkov

8 hours ago

A standard transformer keeps every past token in memory and looks back at them through attention whenever it needs to, so it never has a reason to boil down its history into a small summary that it updates one step at a time the way a recurrent network does. This freedom helps the model fit its training data, but it often leaves the model's internal representations disorganized, so the model fails on inputs different from those it trained on. The Manhattan taxi example considered in the paper demonstrates this: a model trained only on turn-by-turn routes learns to predict legal turns correctly, yet when researchers reconstruct the street map the model has effectively built from its own predictions, that map is full of impossible roads. The authors of this paper from Microsoft Research introduce Next-Latent Prediction, an extra training objective that asks the model to predict its own next internal state (the compressed vector it forms after reading each token) from its current state plus the next token, using a small separate network to model that one-step transition. The authors prove that a model satisfying this consistency condition is forced to compress its history into a "belief state," meaning a summary that holds everything needed to predict the future and nothing redundant, and they show this can be done without slowing the transformer's fast parallel training. The helper network also leaves inference untouched, since it is used only during training and discarded afterward. Across taxi-route world modeling, an arithmetic reasoning game, graph path-finding, and full language-model pretraining at the 1.3-billion-parameter scale, the method yields more compact internal representations, better planning, and a useful side effect for speed: because the small transition network can be run forward on its own to guess several upcoming tokens, the model can verify multiple tokens at once and run inference up to about three times faster. chapterpal.com/s/6d62f92c/nex…

2 3 19 1K 5

View Details

BURKOV @burkov

15 hours ago

Since they took the stillborn Metaverse from Zuck, all he does at Meta looks like sabotage.

Polymarket @Polymarket

21 hours ago

NEW: Meta has reportedly paused its AI training program that recorded employees’ keystrokes & mouse movements after an internal leak exposed sensitive data companywide.

84 65 1K 154K 103

3 2 32 4K 3

View Details

BURKOV @burkov

17 hours ago

@RanDomAjy @elonmusk It's not supposed the be this way. There's no technical difficulty in fixing this.

1 0 3 257 0

View Details

BURKOV @burkov

18 hours ago

Hey, @elonmusk, could you please address this injustice on X already? When someone replies to your post or comment with an insult and then blocks you so you cannot tell them who they are, it's unfair. It's my comment, and they replied to it. I must be able to reply. Otherwise, it's like kicking someone in the back and hiding behind a closed door. Unfair.

10 1 59 4K 1

View Details

BURKOV @burkov

18 hours ago

Childish argument. First of all, we all know there are plenty of stupid permits and limitations in Europe, take Germany for example. Second, we all know that many countries do have rules forbidding lowering the indoor temperature below some ridiculous threshold. Third, you don't need to force plebs to do something. You can simply make it too expensive or you can shame them and the plebs will choose to do that on their own.

3 0 3 182 1

View Details

BURKOV @burkov

21 hours ago

Only about 25% of French households have air conditioners. Not that I'm shocked, but only because I already knew that European citizens are forced to take a shower only a couple of times a week and wear sweaters at home in the winter. No sane person without the Stockholm syndrome would accept living in such conditions in 2026. European bureaucrats and left-wing politicians are working and living in air-conditioned spaces, but the average European citizen lives in a medieval castle. All hotels I ever stayed in Europe have air conditioners, because why would anyone respect the aborigine, right? npr.org/2025/08/19/nx-…

21 3 39 5K 4

View Details

BURKOV @burkov

18 hours ago

@ACEaceACE4ce I would not fact-check a bot, especially when its comment is irrelevant. Not having an air conditioner in 2026 and not being able to have one is outrageous enough on its own.

0 0 1 120 0

View Details

BURKOV @burkov

18 hours ago

@_DataStrategies There are various ways to force people into doing something. Making something too expensive or shaming for doing something are two such ways. ec.europa.eu/eurostat/fr/we… group.vattenfall.com/press-and-medi…

1 0 1 143 0

View Details

BURKOV @burkov

18 hours ago

@RalphAichinger Not really. It might sting depending on the wool. I prefer pajamas.

0 0 0 45 0

View Details

BURKOV @burkov

19 hours ago

@zcserei Fart anything once again and it will be your last fart here.

1 0 0 39 0

View Details

BURKOV @burkov

19 hours ago

@RalphAichinger Indeed. Why also not wearing ski pants and boots indoors?

2 0 2 137 0

View Details

BURKOV @burkov

19 hours ago

@zcserei I have data, and you, dumbass? ec.europa.eu/eurostat/fr/we… group.vattenfall.com/press-and-medi…

0 0 0 46 0

View Details

BURKOV @burkov

19 hours ago

@GaryMarcus They are most likely cooking something awesome with the World First AI Software Engineer, Devin 😆

0 0 5 662 0

View Details

BURKOV @burkov

19 hours ago

The Sakana Fugu technical report: chapterpal.com/s/7ff4f6ba/sak…

0 2 7 1K 7

View Details

BURKOV @burkov

19 hours ago

My take on Sakana Fugu is available to Cool Supporters on Substack, Patreon, and Buy Me a Coffee: aiweekly.substack.com/p/on-sakana-fu… patreon.com/burkov/posts/o… buymeacoffee.com/burkov/on-saka… Also, I just published a couple of days ago my take on GLM-5.2. Become a Cool Supported and stay informed about what you should care about and get other perks such as early access to new book chapters and signed books.

1 0 6 1K 1

View Details

BURKOV @burkov

a day ago

@atomic_chat_hq "surprisingly performed near GLM 5.2." It is "orchestrating" frontier closed-weight models and cannot even beat them. It's an embarrassment.

1 0 31 2K 2

View Details

BURKOV @burkov

a day ago

Without the Soviet Union, the workers would still be sleeping on the ropes.

1 0 23 2K 3

View Details

BURKOV @burkov

a day ago

Free and open frontier AI is the first truly communist gift China has given to humanity. It's the first such gift since the Soviet Union gave the workers of the world gender equality, an 8-hour workday, a month of vacation, and social security. Without China, three fat U.S. capitalists would still be selling us GPT-3.7 with ads.