It is probably the first-time aha moment the author is talking about. But under the hood, it is probably not as magical as it appears to be.
Suppose you prompted the underlying LLM with "You are an expert reviewer in..." and a bunch of instructions followed by the paper. LLM knows from the training that 'expert reviewer' is an important term (skipping over and oversimplifying here) and my response should be framed as what I know an expert reviewer would write. LLMs are good at picking up (or copying) the patterns of response, but the underlying layer that evaluates things against a structural and logical understanding is missing. So, in corner cases, you get responses that are framed impressively but do not contain any meaningful inputs. This trait makes LLMs great at demos but weak at consistently finding novel interesting things.
If the above is true, the author will find after several reviews that the agent they use keeps picking up on the same/similar things (collapsed behavior that makes it good at coding type tasks) and is blind to some other obvious things it should have picked up on. This is not a criticism, many humans are often just as collapsed in their 'reasoning'.
LLMs are good at 8 out of 10 tasks, but you don't know which 8.
It simply forces the model to adopt an output style known to conduce systematic thinking without actually thinking. At no point has it through through the thing (unless there are separate thinking tokens)
This highlights an important limitation of the current "AI" - the lack of a measured response. The bot decides to do something based on something the LLM saw in the training data, quickly u-turns on it (check the some hours later post https://crabby-rathbun.github.io/mjrathbun-website/blog/post...) because none of those acts are coming from an internal world-model or grounded reasoning, it is bot see, bot do.
I am sure all of us have had anecdotal experiences where you ask the agent to do something high-stakes and it starts acting haphazardly in a manner no human would ever act. This is what makes me think that the current wave of AI is task automation more than measured, appropriate reactions, perhaps because most of those happen as a mental process and are not part of training data.
I think what your getting at is basically the idea that LLMs will never be "intelligent" in any meaningful sense of the word. They're extremely effective token prediction algorithms, and they seem to be confirming that intelligence isn't dependent solely on predicting the next token.
Lacking measured responses is much the same as lacking consistent principles or defining ones own goals. Those are all fundamentally different than predicting what comes next in a few thousand or even a million token long chain of context.
Indeed. One could argue that the LLMs will keep on improving and they would be correct. But they would not improve in ways that make them a good independent agent safe for real world. Richard Sutton got a lot of disagreeing comments when he said on Dwarkesh Patel podcast that LLMs are not bitter-lesson (https://en.wikipedia.org/wiki/Bitter_lesson) pilled. I believe he is right. His argument being, any technique that relies on human generated data is bound to have limitations and issues that get harder and harder to maintain/scale over time (as opposed to bitter lesson pilled approaches that learn truly first hand from feedback)
I disagree with Sutton that a main issue is using human generated data. We humans are trained on that and we don't run into such issues.
I expect the problem is more structural to how the LLMs, and other ML approaches, actually work. Being disembodied algorithms trying to break all knowledge down to a complex web of probabilities, and assuming that anything predicting based only on those quantified data, seems hugely limiting and at odds with how human intelligence seems to work.
Sutton actually argues that we do not train on data, we train on experiences. We try things and see what works when/where and formulate views based on that. But I agree with your later point about training such a way is hugely limiting, a limit not faced by humans
Someone arguing that LLMs will keep improving may be putting too much weight behind expecting a trend to continue, but that wouldn't make them a gullible sucker.
I'd argue that LLMs have gotten noticeably better at certain tasks every 6-12 months for the last few years. The idea that we are at the exact point where that trend stops and they get no better seems harder to believe.
One recent link on HN said that they double in quality every 7 months. (Kind of like Moore's Law.) I wouldn't expect that to go forever! I will admit that AI images aren't putting in 6 fingers, and AI code generation suddenly has gotten a lot better for me since I got access to Claude.
I think we're at a point where the only thing we can reliably predict is that some kind of change will happen. (And that we'll laugh at the people who behave like AI is the 2nd coming of Jesus.)
Anthropic is leaning into agentic coding and heavily so. It makes sense to use swe verified as their main benchmark. It is also the one benchmark Google did not get the top spot last week. Claude remains king that's all that matters here.
Unrelated to the paper, the most satisfying (would-be) explanation of gravity I ever learned was from AdS/CFT correspondence. Basically, it says that a Conformal Field Theory in D dimension is equivalent to a theory with gravity in D+1 dimensions. In other words, the physics that describe the 2D boundary of a anti-de Sitter space corresponds to theories that describe gravity inside that 3D space. This would click well with the holographic universe theory and entirely eliminate the problem of needing to explain gravity since it would be an emergent force
But alas, our universe is more like dS space (positive curvature), not AdS (negative curvature).
I have felt like a perennial browser refugee for a while. For about 20 years now (since OG Firefox was at peak and Chrome was not yet launched), every new browser promises the same things, gets popular enough, then does a full or partial 180.
While I like the pitch of this browser, I find it a little difficult to take it at the face value, especially given there is no info on the founders, or whether it is run as a company or a non-profit etc.
Perhaps someone in this thread could answer: which company/org structure provides best guarantees against gradual, slow, multi-year rot that seems to take over everything?
I would happily pay a small monthly subscription fee for a browser if it has strong legally protected privacy guarantees.
I feel the same. For now, I've made peace with having to switch to "whatever is the latest maintained fork with privacy defaults" every 6 months. Hopefully Ladybird becomes a usable browser sometime soon.
The hard answer is the project cannot attract the good engineers anymore because it eventually stops being a growth project. Without being a growth project, you don't get investment into what you want to do anymore and there is less potential growth in your career and income.
Browsers are always going to be "as-is, best effort" . No one, not even google is going to stick out their neck and protect your privacy, that's up to you, and especially not "legally" as that has aspects of easily being sued when privacy/money is involved. Certainly not for a "small monthly fee". Best you're going to get is open source and security community scrutiny of said open source code
It's hard to predict what future generations of developers are doing, but right now Ladybird seems to have the right values embedded into their nonprofit structure.
All the other browser projects have to be enshittified eventually, and therefore have to fulfill other interests than their users' interests to get there.
For what it's worth, I like Orion - built by the same team that built the Kagi search engine. It's a shit browser for developers (inspect panel crashes half the time and other bugs), but I trust it way more than Chrome or even Safari. For development tasks - if I need to, I simply switch to Firefox.
reply