Is it possible someone on the team published a blog post about why it wasn't possible? Or that otherwise the AI learned from your 5 years of effort? Some of the code made it on GitHub or something?
It's totally plausible that it didn't & it still got it right, I'm just curious.
It's beside the point and so I only note it out of interest, but the Chinese writing system doesn't use an alphabet (or a syllabary like Japanese kana), it's logography.
Who have you ever heard make a sincere, good faith argument that a toaster is conscious? Who do you imagine would argue against you if you asserted they were mere automata?
If you can't identify anyone, then this analogy doesn't work.
> Who have you ever heard make a sincere, good faith argument that a toaster is conscious?
More than one, for many classes of devices, incl. toasters. Some were drunk, some were insane, and some were delusional.
LLMs are no different. They are automata, yet delusional people bring out pitchforks and torches when someone points out that they are just statistical models, and they don't even work when there's no input to them.
> they don't even work when there's no input to them.
Why is that needed for consciousness? They're artificial. If we put someone in a coma, they don't talk, they're not really conscious. Just because the AI model has a more obvious off switch shouldn't make a difference. It's easy to imagine gluing a cron job to the model so it works randomly. If that doesn't count because it's external, if we take a brain out of a human's head and slap it on the table, it's not going to do anything. If I take the AI's model file and stick it on a USB flash drive, it doesn't do anything. Without a computer to run it on, it doesn't do anything, just like a human brain doesn't do anything without the rest of the human body as the harness.
The underlying question is, since we created LLMs, we can see into the actual matrix math, the linear algebra that comprises them. So it's easy to dismiss them as a next word guesser. How could consciousness arise from guessing the next word? But we don't know where consciousness comes from in the first place!
So since we don't know, the fact that, yes, the anthropomorphically named LLM "neurons" are merely matrixes of numbers and we do linear algebra on them; yes, that gives us much more insight as to what it's doing internally compared to a human or any other lifeform with consciousness's brain. And yes, human neurons are much more complicated than a mere 2d matrix of numbers, so far as we know. But we don't know!
The indightment of LLMs is that they can't say they don't know, and would prefer, instead, to hallucinate and bullshit an answer instead. They can't help it, they learned from the best. As a human though, I don't know if they're conscious, and what I'm going to say about that.
Their being statistical models and their being conscious are not contradictory unless proven otherwise. That's not knowledge, it is assumption.
It would appear to me you have no interest in a real, good faith discussion on this topic because you think anyone who disagrees with you is necessarily delusional. Which is a shame, and that's the kind of dogma you are criticizing.
This was exactly the point of the story, it's too uncomfortable to admit that we don't know what consciousness is and what is and isn't conscious, so we just brush it under the rug.
Sorry, I'll be blunt: Are you sure you are not projecting?
My perspective comes from a set of pillars. First, I work at an HPC center, where we support running and development of AI systems, incl. international projects. IOW, I have knowledge about how these systems built, work and needs to continue working.
Moreover, I'm an HPC programmer myself, so I'm not completely uninformed about the math this involving this thing, and I'm lucky enough to have friends who are much more dedicated than me, and we discuss how this thing works and feels like this way.
I'm not an AI hater per se, being programmed AI systems in the past, incl. emergent intelligence systems with multi-agents which can span continents if need be (this was my master's thesis, time flies).
However, knowing what these things are capable of and how they are built. I don't believe them they're conscious/sentient beings. I also had much more time to ponder on these things even before LLMs being a thing. Some hard sci-fi books have asked these questions seriously in their captive adventures way earlier. If one reads these books seriously, there are a lot of philosophical angles to consider and draw upon.
I can discuss in good faith. For hours, days or months even, but throwing "you're a narrow-minded dogmatic luddite neanderthal!" card to anyone disagreeing with you is not it.
Positive, yes. I never called you a luddite or a neanderthal or anything of the kind.
It's perfectly fine to believe they are not conscious, I am not convinced they are, but asserting anyone who disagrees with you is delusional is unfortunate.
From my perspective, you inadvertently implied, but no offense taken, all is well.
No, I didn't assert anything. I have just given examples rooted in my experience.
None of my friends who also happen to know how these things work told or defended that they are conscious, even intelligent. Maybe my friends are dumb, I dunno.
Once I have seen a man who claimed that evil has possessed the POS device at his desk. The thing was printing "cannot connect to server" on the receipt printer every 10 minutes, yet he didn't know how that thing worked, and was a bit too high to read the paper the thing was printing out.
This age's LLM craze is akin to "wonder inventions" of 70s, which are deemed dangerous or harmful in the future. LLMs will be with us, but we need to pass beyond the hype and stop sweeping the problems they create (environmental and societal) under the proverbial rug.
If you didn't mean to say that people who believe AI is conscious are delusional, then I don't understand how to read your comment. If you're interested in a good faith conversation, I'm very confused why, when I asked if anyone sincerely argued that toasters are conscious, you brought up people who were drunk or delusional.
It's a two way street. I might have failed to convey my point clearly, too. While I'm fluent, this is not my native language. Some edge cases in meaning still betrays me and I make mistakes I didn't intend doing.
I don't think you understood the point of the story. It's not that LLMs or agents are conscious, it's that our dismissal of the possibility is reflexive and uninformed. Personally I think anyone who has made their mind up about whether or not LLMs/agents are or can be conscious has done so before the evidence is in.
The story does not assert that search and replacing "meat" with "weight" makes them conscious through some magical mechanism. It's a thought experiment.
No, it's a story. Stories can be about real things (the original story is about a real consciousness, as shown by it being produced by that consciousness), but they aren't automatically just because they're good stories.
You're falling into the trap of assuming that a good presentation--being convincing--is the same as being truthful.
You can't do the same with a toaster. Physically you could write that story. But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness. You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
You could absolutely write a compelling story about a sentient toaster; it's been done before [1].
That is entirely separate to whether or not it would be a meaningful way to understand the world; a convincing story is not the same thing as one that is true.
I never said you couldn't write any arbitrary compelling story about a toaster, I said that this specific hypothetical story, where you rewrite "They're made of meat!" to be about a toaster, would not be compelling.
I am doing my best to communicate with you but to be honest you are not hearing me (across both responses), and I am out of words.
Just wanted to say, I appreciate your patience and good sense in this thread.
It's difficult to tell who's trolling -- probably best to go with the charitable assumption that everyone is honestly trying to convey their opinion, but mostly talking past each other. Unfortunately these discussions about the nature of consciousness never go anywhere useful.
I think I'm probably in the same boat as you, roughly: a) LLMs are doing something really interesting that resembles in many ways both intelligence and consciousness; b) I suspect they're not actually conscious but I don't know how you'd know for sure; c) it all just drives home that we still don't really know what consciousness actually is. But like (a), it's definitely something really interesting...
I read some stuff once where the author argued that almost everything takes in values, performs operation on them and gives output. Some things also store things. Then, if everything is doing logic operations there is no telling where intelligent "life" might be hiding. It also is really one giant system.
The toaster has hard coded or configurable weights. It makes a product from heat and time.
You could make toast by heating a brick on a campfire. It would be a clear sign of intelligence.
If we lift one weight out of your brain we wouldn't look at the number and say it is intelligence. It must exist in a chain or matrix multiplication to qualify.
The fun thing is that the timer and feedback mechanism do exist in a chain of events.
It's part of the system and it takes all steps to complete the task.
Our chain of thought that leads to making toast would be abandoned if it wasn't in working order.
It would be equally compelling, because the compelling nature of the story comes from the language, the presentation, rather than the [specific thing being ascribed consciousness].
No, the original "they're made out of meat" works because we're confident that we are in fact intelligent and conscious, despite how ridiculous and unlikely the author manages to make it sound.
"They're made out of weights" works precisely because LLMs really do have this mysterious property that they seem somehow intelligent even though nobody can explain exactly why, and there's active debate over whether they could be considered conscious.
The thing being discussed isn't simply an arbitrary MacGuffin; in both cases the nature of the thing is central to the impact of the story.
I disagree; it works in the original because it's the unlikely consciousness that produces the text itself; in the LLM case, it's produced by the likely consciousness.
"Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it.
> Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it
This is well put. We don't need to imagine how a human views a llm because we can ... just do that. Everyone capable of reading the story is also capable of thinking about how they feel abouy llms that exist right now and you've probably used.
The trick of the original story is inverting your perspective, moving your view point fron yourself to an "other" (which I think is a primary qualifier for most good fiction).
But that toaster would just be a device to talk about consciousness in general.
In this case it does that and also it talks specifically of the LLM case, which can spark the discussion. Unless you believe to have the only valid and true opinion on the matter, and affirm that a normal toaster is just the same as an LLM in this topic.
I agree that is the mode of argument; reductio ad absurdum is a brittle argument, because it only works if the analogy holds. I argued the analogy doesn't hold.
> But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness.
Teapots are not compelling.
> You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.
God is compelling t billions of people.
Is Russel’s Teapot a bad argument in the God debate?
> Is Russel’s Teapot a bad argument in the God debate?
What's the relevance? If the argument made here are was a good argument, it wouldn't matter if Russell's argument was bad. We could construct a bad argument using reductio ad absurdum right here and now and it wouldn't matter to either argument.
Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
> Can you be straight with me? You know the salient difference between asserting the consciousness of a toaster and the consciousness of an AI, right? It isn't a mystery to you why we would find one line of inquiry interesting and the other not so much?
There are two aspects here.
1. That people find the question interesting
2. That it has any bearing on reality (ontology?)
The first aspect is anthropology. Russel’s Teapot is not supposed to undercut any anthropological arguments. It’s supposed to undercut the second aspect.
So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
> For instance, it's probably a real possibility in your mind that I am not a human and am an AI. But you probably aren't entertaining the hypothesis that I'm a toaster.
Yeah. AIs know how to use computers. What’s this got to do with consciousness? Whether or not you are an AI is practical and disprovable. Consciousness is so ephemeral (for lack of a better word, not literally) that Philosophical Zombies is a real argument/thought experiment.
You may think I’m being coy (“Can you be straight with me”) but that’s not my intent at all.
Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument. I don't see any relevant connection. Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
> So far you have said that the argument is compelling. What’s that got to do with reality? A robot cow could be sexually arousing to a real bull.
AI is a real phenomenon that we can study and measure. There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
To put a fine point on it, it appears on it's face that AIs could be conscious. They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing. So at face value, the burden of proof is on them not being conscious. Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
> Much like Russel argued that the burden of proof of God's existence is on theists, the burden to establish this parallel is on you as the person forwarding the argument.
I pointed out the parallel in both statements. I can't do more than that.
> Russel isn't arguing that a teapot is as real as God in the same way it's disputed here that a toaster is as conscious as an LLM.
The teapot isn't real and the toaster consciousness is not real. What am I missing?
> AI is a real phenomenon that we can study and measure.
Robot cows are real as well.
> There is no experiment that anyone has devised can determine whether or not they are conscious, so that is the reality - uncertainty. That doesn't mean they're conscious. It means that the belief they are not conscious is assumption.
Yeah. You can't prove it for any entity. I agree.
> You might say the same of a toaster, but these hypothesis are not equally strong. The toaster doesn't exhibit any behaviors to suggest that it's conscious. Consciousness isn't a hypothesis with any explanatory power for the observed behaviors of a toaster. It's not a hypothesis that's on the table. That's why the analogy doesn't work.
The bull swears that the robot cow is a real cow. But we know better.
> To put a fine point on it, it appears on it's face that AIs could be conscious.
It doesn't to me. Not any facelength.
> They can put on a very convincing performance of being a person. A sufficiently convincing performance is indistinguishable from the real thing.
Objective reality has never cared (am I anthropomorphizing now?) what is indistinguishable for people.
> So at face value, the burden of proof is on them not being conscious.
Which party is the burden of proof on? This is confusing since you are saying that the burden of proof is on a position (on them not being conscious).
Is the burden of proof on people who argue that they are n o t conscious? That's peculiar.
I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
We obviously can't demand a falsifiable theory here. But we have to do better than arguing from incredulity.
> Reductionist arguments that present the mechanics of how they work and leap to their not being conscious don't work, because there is no law saying a statistical model can't be conscious. That's an assumption, not knowledge.
They don't have to rise to the level of disproving something for which they have no burden to disprove.
Would you agree, for the sake of argument, that it's more interesting to discus chatgpt 5.5's similarity to sentient/sapient/conscious than, say, my mechanical toaster?
> I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
Also, you say this like "duck" isn't an arbitrary, artificial category created by humans, a "map not the reality" if you will.
There's very few things that humans can understand to the level of putting them into truly objective scientific categories (various pure elements maybe?), everything else we more or less bodge together for the sake of getting on with life.
It's not like conscious has some kind of formal objective provable definition, even inside the world of human created language and terms.
As far as I know, in the real world, if something looks enough like a duck (and can breed with a duck maybe) we, humans, do call it a duck.
> I pointed out the parallel in both statements. I can't do more than that.
You mean these statements?
No one cares about teapots in space either (Russel).
Teapots are not compelling.
I guess you did but that's pretty much leaving me breadcrumbs and expecting me to make your argument for you. It seemed to me like you were talking about reductio ad absurdum with that argument as an illustrative example. Perhaps you overestimate my cleverness.
> Which party is the burden of proof on? This is confusing since you are saying that the burden of proof is on a position (on them not being conscious).
It's metonymy. "X" stands in for "people who argue for X". May I ask if you were sincerely confused? You told me you aren't being coy but I have a hard time believing that this was so unclear.
> I have never heard about any principle in philosophy or in science that says that, given enough Looks Like A Duck points, it is a duck. Based on subjective experience, even.
If it looks like a duck, if it quacks like a duck, you should adjust your priors to assign a higher likelihood that it is a duck. All measurements contain error; you can't ever observe, "that is a duck," only "that looks like a duck". All knowledge is founded on a sufficiently deep stack of "looking like a duck" that we may assert it with confidence.
To the extent we have objective measures (like conducting a Turing test on blind participants), it can meet them too. You can't say the same of a toaster.
> We obviously can't demand a falsifiable theory here. But we have to do better than arguing from incredulity.
"It is a statistical model, ergo it is not conscious" is also an argument from incredulity. I don't know if that's your view or not but it's the one my remarks have been addressing in general.
Just so you know, I have nothing to do with Stanford, but I am flagging this as conspiratorial nonsense. So when you comment is flagged, I just want you to know that it doesn't confirm your belief, it's just that this comment harms discussion and so must be removed.
Yes, mea culpa. Occasionally I break that rule on my own judgement. Feel free to flag my comment. (I think it's important to disconfirm conspiracy theories.)
for what it's worth I have no idea why it would be nonsense to question institutional motivations especially in the context of an academic article that could easily be corporate propaganda, I also think that shutting conversations down is much more harmful than discussing topics that are potentially harmful
Completely unevidenced conspiracy theories can only harm the discussion. The only possible benefit is to disconfirm conspiracy theories and discourage paranoid thinking. The odds that Standford as an institution are astroturfing on HN round down to 0.
What they're almost certainly observing is that these critical comments are being flagged as inappropriate. People make inappropriate comments that happen to contain criticism all the time, and I frequently see people edit them to declare that they were flagged because the group they're criticizing is astroturfing. It's virtually never the case. I've never seen it happen.
But to be clear I am completely ambivalent on Stanford and if you want to criticize them, more power to you.
may I ask why you effectively said 'conversation over due to harm reasons' instead of asking for evidence to support the conclusion that you believe is not possible? I don't see why it is inherently harmful to discuss the seemingly impossible. I also don't see why it's relevant to bring up your n=1 sample (although it is as relevant as my n=1 sample, which has plenty of astroturfing witnessing [unspecific to Stanford])
> ALL context/prompt is instructions, there is no data. It is just unsolvable, period.
That really isn't true. There's no law of physics preventing you from having separate data and instruction inputs to models. The model's transcript format generally distinguishes between prompts and instructions and tool output and such. This isn't a solved problem, and it's possible it's entire unsolvable, but it probably is possible (in general, not with current models) to reject prompt injection to several nines.
This is a lot like making the same statement about CPUs, "the von Neumann architecture doesn't distinguish between code and data so it's impossible to reject malicious instructions." There's actually a lot you can do to reject malicious instructions, you can prevent execution in certain pages, you can prevent certain privileged instructions from being executed in certain pages, you can employ stack cookies, et cetera. Do they prevent all exploitation in all circumstances? No. But each component does function in it's lane and it is possible to create programs with high (though not absolute) guarantees against unauthorized code execution by composing them.
Similarly, you could prevent certain tokens from appearing in the prompt portions of a transcript, you can have a model with multiple input heads only one of which is trusted, etc. I'm not saying those techniques will necessarily work, but it is more complex than "models can only possibly take a single and undifferentiated input stream".
A lot of the solutions in the CPU space involve things like memory allocation flags, NX bits, canaries, etc. that fire deterministically. Those things are fundamentally not applicable to LLMs, and without those things modern software would be in a vastly worse place.
You could imagine that there are things to change around LLM architecture that will improve its ability to reject prompt "injection", but I think it's fundamentally true that from an information theory perspective there's no bright line between "instruction" and "input data" possible.
Nondeterminism is a red herring. There is a bright line between instructions and data right now, in virtually every transcript format. That we have not succeeded in training an LLM to respect it to a very high degree doesn't imply it is impossible; that they are nondeterministic doesn't imply it is impossible; only that we won't succeed 100% of the time.
A cosmic ray (or rowhammer attack) could flip an X bit too, there isn't anything truly deterministic under the sun.
I read the misanthropy as ironic. They're applying the same reductionist logic to humans, not because they are misanthropic, but to illustrate that it doesn't help us understand the case we can all agree on. "Humans aren't sentient either" is definitely not the takeaway.
The point is, we have no idea what "sentient" or "intelligent" even means. If we agreed on the definitions, the debate would have been settled long ago.
Let's assume we have infinite memory with constant time lookups. With a sufficiently large lookup table, you could exactly replicate the behavior of any person. You could encode it as a next-token predictor: you have precomputed every possible prefix and assigned it a next token. This is a Chinese room, but it is completely indistinguishable from an intelligent, sentient person. There is no experiment you can design to slip a piece of paper (a prompt) under the door to determine whether it is Bob or the lookup table clone of Bob inside the room.
Does that make the lookup table conscious or alive? Undefined. It's the wrong question. Or it's not a question science can address.
So we cannot dismiss on it's face the idea that next token predictors "are not and never will be alive" unless by "alive" you simply mean "biological," but that's not really what's debatable.
The argument is also very brittle because they are not in fact all next token predictors. I doubt people making this argument would be willing to concede that diffusion models are more likely to be conscious than causal models (which I do not believe but is an implication of the argument).
I'm not saying that they are conscious or sentient to be clear, but the reductionist argument that they are next token predictors and therefore don't have some property humans have is not an argument. That's going from A directly to Z. You need to flesh out the bit in the middle because that doesn't follow.
Right. Humans are a biological computer. They have a state and they compute an output. I had to look this up (and use AI) but an estimate for the state of a human mind is about 5 peta-bits (10^15) and the estimated processing power is about 1 exa-FLOP (10^18). Compare this to the largest models at ~5 tera-bits (10^12) of state space and ~2 x 10^14 FLOPS (for one session with some reasonable token rate).
Assuming the above is anywhere near true (I think there's a lot of debate about the capacity of the human mind, where data is actually stored, and where compute happens) then we are talking about 3 orders of magnitude win for humans in state and 4 orders of magnitude in compute. And we're doing all that pretty energy efficient as well.
The other big difference in humans is that we learn and the model only "learns" in context. Out "learn" space is much larger than the 1M tokens that frontier models struggle with.
Anyways, point is that a computer can appear to be alive. If we simulate the human brain perfectly and train it like a human then we'll have something that has human capabilities. LLMs have interesting capabilities but at least at this point not fully human ones (and the delta-state/compute would be a hint that there is still a large gap to cover).
human context/memory could just be an Agents.md file too that gets read instantly before your next token prediction runs. The AI can make multiple such memory files and read on demand depending on what the topic is, kind of like how as a human when you try to remember a math problem you don't go to your childhood bicycling Agents.md file either.
It's totally plausible that it didn't & it still got it right, I'm just curious.
reply