dev0panda @dev0panda

@splitbycomma See I would be in the same boat as you but I just don’t see why superintelligent machines would care about us? What makes you think future AI will be safe or care about us? It’s still an unsolved problem and nobody understands how models work internally

@eggynack @gayestgaymer1 @biggestjoel @artchad Like, I’m not arguing whatsoever against having policy to regulate current harms posed by these systems, but when all the data points to the degree of harms getting worse fast, why not just have even more strong policy? seems like an easy position to move to from yours.

@eggynack @gayestgaymer1 @biggestjoel @artchad But also you’re absolutely right that there’s a misalignment between you and Google too!! all the more reason to be super proactive with policy, there’s so much here that is driven by profit incentive at the extreme detriment to society.

@eggynack @gayestgaymer1 @biggestjoel @artchad As an example, evolution has the optimization function of “inclusive genetic fitness” but our inner optimization barely cares about this. We are misaligned with this with things like cheesecake, contraceptives, and genetic modification. AIs would have their own versions of these

@eggynack @gayestgaymer1 @biggestjoel @artchad Actually, the important bit is that the outer optimization process does not reproduce the same inner optimization process. Great explanation by Rob Miles here: youtu.be/bJLcIBixGj8?si…

@eggynack @gayestgaymer1 @biggestjoel @artchad Like, it just absolutely is *not* “duplicating what it’s seen humans do”, these capabilities arise because AIs are learning beyond simple replication of training data. I don’t see how new models finding zero-day cybersecurity exploits isn’t an increase in “thinking” ability

@eggynack @gayestgaymer1 @biggestjoel @artchad How does it solve unsolved Erdos problems then? Or how are LLMs able to play games they weren’t trained on better than human novices? or how can they do office work better than some humans already? The idea that “it’s not *really* thinking” doesn’t really matter here

@eggynack @gayestgaymer1 @biggestjoel @artchad There’s no reason any step of this requires consciousness! it just gets smarter. If it does somehow produce consciousness, that’s interesting, but not relevant to this discussion

@eggynack @gayestgaymer1 @biggestjoel @artchad in the same way that a Go AI can beat human experts *every time* without being conscious, and in the same way that LLMs can beat human experts *every time* in raw manipulation of text, more generally capable AIs can beat humanity *every time* at the task of “world manipulation”

@eggynack @gayestgaymer1 @biggestjoel @artchad this is just avoiding what we mean by “goal” here — a system just “wants” the thing it tends to do. A thermostat “wants” to keep the temperature in the right range in the same way that an LLM “wants” to produce text, but it doesn’t matter if there’s “intention” — it just does it

@eggynack @gayestgaymer1 @biggestjoel @artchad nobody is claiming “consciousness” — but also, even if you thought that, the idea that you can get consciousness by just making neuron count go up is surprising in human brains too. Intelligence is decoupled and we can have very smart stuff without it being conscious.

@eggynack @gayestgaymer1 @biggestjoel @artchad systems are *literally trained to pursue a goal* that is somewhere along the lines of “predict text based on the internet”, “predict text that makes humans thumbs up your response”, and “predict text that solves math problems” through the pre-training and training processes

@eggynack @gayestgaymer1 @biggestjoel @artchad What do you think AI is doing? How is it not intelligent? LLMs think in alien ways and are less capable overall than humans right now, but if you think they’re not intelligent, I would ask you to define “intelligence”

@eggynack @gayestgaymer1 @biggestjoel @artchad Adding this graph again. If you know about “scaling laws” you would understand why there was a massive jump from GPT-2 to GPT-4 in capabilities, or why Claude Mythos is capable of cybersecurity risks. We have every reason to think it will get more capable.

@eggynack @gayestgaymer1 @biggestjoel @artchad The most strong version of what is being claimed is just that AI will kill everyone, regardless of methodology. What’s your counterargument to instrumental convergence? how is that not a basis to think misaligned ASI will do this?

@eggynack @gayestgaymer1 @biggestjoel @artchad Also, from metr.org: The thing is that the more scary level of capabilities will come fast if they come at all, so policy has to be extremely preemptive here. And yet almost nothing has been done in regulation of this technology + companies can’t even align it!

@eggynack @biggestjoel @gayestgaymer1 @artchad Okay, I see the misunderstanding. My intention with that word was just to mean a system with *any* level of capabilities, but as is mentioned in your replies here, capabilities won’t be static anyway by a long shot. I don’t know what “no goals at all” is supposed to mean.

@eggynack @biggestjoel @gayestgaymer1 @artchad What I’m trying to get at is that all these risks are real and should be addressed too, but that’s not happening without good policy, and where all this ends is we would expect to see systems smarter than us. So we should make them safe *before* that, which requires a treaty!

@eggynack @biggestjoel @gayestgaymer1 @artchad I mean, these just appear to be another case of misalignment. We would want an AI system to not do those things since they’re against human interests, and those people seeking bad stuff from AI are either incapable of seeing why it’s bad or just bad people themselves