> LLMs can never become out-of-control superintelligence
The "never superintelligence" part I'll buy, though only in the sense of sample efficiency and generalisation ("quality superintelligence"), as they clearly have a superhuman breadth of skills, and run at superhuman speed.
"Never" out-of-control is obviously falsified by the already existing headlines about times they've gone out of control… in part, in some cases, because of their superhuman speed.
If you're asleep at the wheel, you're legally responsible for the car crash, but the car itself was the thing which by crashing caused injuries.
If you deliberately relinquished control of your computer to OpenClaw, you're (I hope) legally responsible for whatever it does, but that doesn't stop it bankrupting you or deleting all your emails or whatever it was you connected it to.
DNA printers exist and are a thing. In 2010 we could all tell ourselves that no sane person would ever let some future AGI "out of the box" and onto the internet. Today, with the benefit of hindsight, do you seriously expect nobody to connect an LLM to a DNA printer, despite this being a terrible idea, given all the other things they've connected LLMs to despite it being a terrible idea?
That's absurd. The distinction is at the heart of the entire discussion.
It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme
> It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme
On the contrary.
The e.g. paperclip maximiser isn't "an AI decides to make paperclips", it is "some idiot tells an AI to make paperclips and leaves it unsupervised".
Even when people were priding themselves on plain just not believing Yudkowsky's claims that him role-playing as an AI could talk people into letting him out of the box*, the entire point was "let's get an AI to do work so we don't have to".
The entire point of AI has always been to automate stuff, to let ourselves not have to think about the stuff it does. Same as industrial machinery, and it took us long enough to sort out workplace health and safety and emergency off-switches for those. Or even more basic things like not having kids dart in and out of the unstoppable moving steam looms while they were in motion.
We are really, really slow at safety for this kind of thing.
Which is going out of control. Something not under control is out of control. If I jump out of a moving car, I deliberately relinquished control of it. It's still out of control. What a silly semantic game.
"LLMs can never become out-of-control superintelligence" might be relevant if there weren't many hundreds of researchers working (at OpenAI, Anthropic and elsewhere) on AI designs not based on the transformer (LLM) architecture.
People are working on lots of things all the time, so far, nothing has approached the efficacy of the transformer architecture.
LLMs didn't emerge by chance, they are the culmination of decades of research intersecting with brute force engineering rigor in a perfect storm of innovation. You're not just going to stumble into an alternative approach by dumping loads of cash into research.
Most of those researchers would disagree with your statement that the next major breakthrough is probably decades in the future.
Many of the insights accumulated in the decades up to now will probably prove useful for creating non-LLM AI, and the researchers can use LLM AI to speed up their research into non-LLM AI.
Vibe coding has made them a lot more common. Before, you'd need to put a lot more effort into making a website that worked like this, and it wasn't worth it for a random post. Now this person's entire website is posts like this, and I've seen many more in the past few months.
But now with vibecoding it feels like the default for articles to have fancy animation, rather than the exception. I guess that by having a fancier presentation it subconsciously legitimizes the content more so you're less likely to critique it as compared to a simple blog post where you pay more attention to the words and can realize that it's very surface level.
It's the inevitable end game. If the models ever become practically useful in a closed loop, there's no other choice except to keep the model private and use it to compete directly with their current enterprise customers.
Not really. It's possible they could, but in practice they cannot. Creating a billion dollar company requires a good idea, good timing, and a lot of luck, the engineers are the least important part.
More than that it takes things like the right social connections, strong marketing, insight into customer demand, infrastructure spend, etc. You can't normally just convert engineering effort into profit in the way implied.
I laughed at brand risk. I think every player in AI is well past the brand risk. Only posturing you see is being too dangerous. And even that is for hype not actually stopping them. Eventually they push the models out despite that. It really is weird market branding wise.
Just goes to show how desperate they are for compute. It's definitely a brand risk for them, but failing to keep the treadmill going is an existential risk.
reply