Do #LLMs have mental states?
-
I am thinking along the lines of Searle - some sense that the LLM “knows” stuff. Representation isn’t equivalent to “knowing” in this framing. Following a complex computational algorithm isn’t the same as knowing.
-
Modern thermostats certainly have similar representation of temperature when compared to what an LLM would have. A sensor’s output would be converted to a digital representation of the ambient temperature and this would be fed to an algorithm for dynamic responses that can also include user preferences and time of day.
-
@icastico ok. So why *wouldn't* 'extracting an underlying model on the basis of a next token prediction task' that allows you to not just master that task with some degree of accuracy but generates representations with sufficient power to causally support drawing an image count as 'knowing'?
what is 'knowing' if no that?
(I'm not, for a second, saying your wrong, I'm just saying I can't currently understand your answer in a way that makes it feel like a sufficient answer to our question) -
re modern thermostat, I stand corrected! ;-).
There are still fundamental differences with respect to that representation. One it is wholly externally imposed (which the LLMs representation of table issues is not) and the uses a thermostat can make of it are fixed (whereas the whole point of foundation models is that this is not the case).
No external agent has (specifically) created the LLMs representation or set up how it is used.
-
This gets back to that term “interpret” - an LLM that makes a mistake doesn’t evaluate and correct based on examination of its own output - it responds to a new prompt indicating an error with the same “fill-in-the-blank” algorithm - perhaps informed by new information (if that was included in the new prompt). Since the LLM doesn’t know what the representation means- it can’t reinterpret its output without guidance from the user. An LLM wouldn’t re-examine its answer and go “oh wait - I forgot to carry the 1” because it doesn’t know what its answer means.
At least that is what I infer from the output I see from these systems.
-
Dimitri Coelho Molloreplied to Ulrike Hahn last edited by
@UlrikeHahn there is quite some dispute about what counts as the mark of the mental, if anything, and similarly for the cognitive. Often something like a belief-desire psychology is considered central for the mental, while it is not so for the cognitive.
I've argued that LLMs do have aboutness regardless of interpretation in this paper, referred to as intrinsic meaning, in case you might be interested: https://arxiv.org/abs/2304.01481
That just means that they have what it takes to form their own reps.
-
@icastico so your sense of "knowing" is more about 'meta-awareness' (in which case young children and non-human species presumably can't 'know') and not about a particular kind of lawful relationship with outcomes?
-
I don’t think I would take it that far. Kids certainly interpret and know before they know they know. If you know what I mean.
-
Ulrike Hahnreplied to Dimitri Coelho Mollo last edited by
"Often something like a belief-desire psychology is considered central for the mental" and there is a slender peg on which to hang things if ever there was
will check out your paper in a moment
-
@icastico how, though - on your version of 'knowing'?
-
Even young children evaluate the result of their intentional acts to see if they were successful and learn from that experience. Their experiences change the algorithm they use the next time based (at least in part) on internal adjustments - not exclusively on external prompts.
-
@UlrikeHahn @philosophy I think when something is capable of representing something it can be perceived of as having a "mental state" I guess, but in the same way that a slime mold might approach food by the shortest possible path
-
Dimitri Coelho Molloreplied to Ulrike Hahn last edited by
@UlrikeHahn yep, this whole debate is rather complex (or muddled), and I think somewhat misguided. We probably should draw finer distinctions between different kinds of cognitive system, rather than care much about marks of this or that vague category.
-
@icastico and why is that not what ChatGPT4o does, for example during RLHF?
-
Ulrike Hahnreplied to Dimitri Coelho Mollo last edited by
@dcm ok, gave your paper a quick skim- it's great!!
so, back to the main theme - if we buy the premise that the representations in GPT4o are, in fact, representations, and thereby have 'aboutness', is there anything you see *in the specific definition* I put up in the OP that would allow one to conclude that LLMs *don't* have mental states?
-
interesting example- I once had a very intense, multi-day email exchange with a PhD student on why I didn't think the framing of slime mold 'making decisions' was meaningful
-
@UlrikeHahn @dcm You might argue that intentionality requires full agency. GPT doesn't really have that (by design). It very likely has intentions when it's executing a task, "local" to that task, but not overarching long term intentions about itself.
You could then argue that an SQL engine has aboutness and local intentionality, so that's not really enough for mental states.
(Not my opinion, btw. Just an argument that could be made. I think my that we are too far from a workable definition)
-
@UlrikeHahn Just so I understand, though, how large of a space are you considering "LLM" to denote? Is GPT-2 in this family, or does it need some "reactive" component? For example, the generate method maybe requires mental states but that's a layer of abstraction on top of the model, a model itself
-
@pbloem @dcm
I definitely see going down the route of agency as the best way to deny LLMs mental states among what is present in the discussion here.But I'm unaware of that having historically been an intrinsic part of the notion (presumably for lack of need), so the question is *why exactly* would we want to make that move now?
-
I tried to short circuit the need for too detailed debate by jumping straight to ChatGPT4o and highlighting some of the features that I think speak to past debate on these issues