this post was submitted on 21 Jun 2026
24 points (90.0% liked)
Asklemmy
54673 readers
234 users here now
A loosely moderated place to ask open-ended questions
If your post meets the following criteria, it's welcome here!
- Open-ended question
- Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
- Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
- Not ad nauseam inducing: please make sure it is a question that would be new to most members
- An actual topic of discussion
Looking for support?
Looking for a community?
- Lemmyverse: community search
- sub.rehab: maps old subreddits to fediverse options, marks official as such
- !lemmy411@lemmy.ca: a community for finding communities
~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~
founded 7 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
A concept that I think is really helpful for interpreting what an LLM does is the concept of a “Chinese room”. The idea is someone slips a piece of paper containing a message in Chinese under the door and inside that room is someone that doesn’t know Chinese following a set of rules for converting characters and numerals into a response based off their syntax. Afterwards the person in the room creates a response and slips it back out under the door. At no point does the person in the room understand the Chinese in the input or in the output, but the person standing outside of the room might believe there is a Chinese speaker inside of the room. This is the same idea with computerized outputs like LLMs. They only provide the illusion of intentionality and don’t actually have an understanding of inputs or their outputs.
What's the difference between converting the characters and numerals into a response based off their syntax vs understanding them? How can you establish, as the person submitting the notes, whether whoever is in the Chinese room is the former or the latter?
I remember reading something about LLMs not being able to learn "x is y" equivalence relations. Can't find it now but limitations like this are what make differences clear between what humans do and what we've managed to teach the neural network (which will be used to iterate and improve the model further, of course)
In the Chinese box analogy, this would be like them knowing cats are considered cute but not whether considered-cute animals include cats (if I remember the limitation type correctly). If you happen to slip the right instructions/questions, something they've seen before or something they're capable of extrapolating, then nothing seems off; but if someone can say in one paragraph that cats are cute but they know of no cute animal, you'd not think they're understanding what they're saying, and so don't really understand the language even if they give you plausible words in all other cases
(For cats it'll work because there's a billion example sentences out there. LLM vendors are also trying to sidestep such problems by having it generate a bunch of tangential text (in which it might happen to regurgitate the tokens it needs to piece together the answer) before answering the prompt, but that's still not being able to apply logic)
Intentionality is the key difference. You can eventually tell a Chinese room’s nature by giving it new variables that it hasn’t encountered before. New problems lead to algorithmic breakdown.
That said, there’s deeper conversations you can have about what consciousness truly is, of course. My personal view is that it requires a level of complexity that we are still very far away from architecturally, and a level of scalability that we may not even be able to support ecologically. This thought experiment is mainly to show you what the inner workings of a computerized process can look like, and works to provide a demystified perspective.