Do #LLMs have mental states?
-
This gets back to that term “interpret” - an LLM that makes a mistake doesn’t evaluate and correct based on examination of its own output - it responds to a new prompt indicating an error with the same “fill-in-the-blank” algorithm - perhaps informed by new information (if that was included in the new prompt). Since the LLM doesn’t know what the representation means- it can’t reinterpret its output without guidance from the user. An LLM wouldn’t re-examine its answer and go “oh wait - I forgot to carry the 1” because it doesn’t know what its answer means.
At least that is what I infer from the output I see from these systems.
-
Dimitri Coelho Molloreplied to Ulrike Hahn last edited by
@UlrikeHahn there is quite some dispute about what counts as the mark of the mental, if anything, and similarly for the cognitive. Often something like a belief-desire psychology is considered central for the mental, while it is not so for the cognitive.
I've argued that LLMs do have aboutness regardless of interpretation in this paper, referred to as intrinsic meaning, in case you might be interested: https://arxiv.org/abs/2304.01481
That just means that they have what it takes to form their own reps.
-
@icastico so your sense of "knowing" is more about 'meta-awareness' (in which case young children and non-human species presumably can't 'know') and not about a particular kind of lawful relationship with outcomes?
-
I don’t think I would take it that far. Kids certainly interpret and know before they know they know. If you know what I mean.
-
Ulrike Hahnreplied to Dimitri Coelho Mollo last edited by
"Often something like a belief-desire psychology is considered central for the mental" and there is a slender peg on which to hang things if ever there was
will check out your paper in a moment
-
@icastico how, though - on your version of 'knowing'?
-
Even young children evaluate the result of their intentional acts to see if they were successful and learn from that experience. Their experiences change the algorithm they use the next time based (at least in part) on internal adjustments - not exclusively on external prompts.
-
@UlrikeHahn @philosophy I think when something is capable of representing something it can be perceived of as having a "mental state" I guess, but in the same way that a slime mold might approach food by the shortest possible path
-
Dimitri Coelho Molloreplied to Ulrike Hahn last edited by
@UlrikeHahn yep, this whole debate is rather complex (or muddled), and I think somewhat misguided. We probably should draw finer distinctions between different kinds of cognitive system, rather than care much about marks of this or that vague category.
-
@icastico and why is that not what ChatGPT4o does, for example during RLHF?
-
Ulrike Hahnreplied to Dimitri Coelho Mollo last edited by
@dcm ok, gave your paper a quick skim- it's great!!
so, back to the main theme - if we buy the premise that the representations in GPT4o are, in fact, representations, and thereby have 'aboutness', is there anything you see *in the specific definition* I put up in the OP that would allow one to conclude that LLMs *don't* have mental states?
-
interesting example- I once had a very intense, multi-day email exchange with a PhD student on why I didn't think the framing of slime mold 'making decisions' was meaningful
-
@UlrikeHahn @dcm You might argue that intentionality requires full agency. GPT doesn't really have that (by design). It very likely has intentions when it's executing a task, "local" to that task, but not overarching long term intentions about itself.
You could then argue that an SQL engine has aboutness and local intentionality, so that's not really enough for mental states.
(Not my opinion, btw. Just an argument that could be made. I think my that we are too far from a workable definition)
-
@UlrikeHahn Just so I understand, though, how large of a space are you considering "LLM" to denote? Is GPT-2 in this family, or does it need some "reactive" component? For example, the generate method maybe requires mental states but that's a layer of abstraction on top of the model, a model itself
-
@pbloem @dcm
I definitely see going down the route of agency as the best way to deny LLMs mental states among what is present in the discussion here.But I'm unaware of that having historically been an intrinsic part of the notion (presumably for lack of need), so the question is *why exactly* would we want to make that move now?
-
I tried to short circuit the need for too detailed debate by jumping straight to ChatGPT4o and highlighting some of the features that I think speak to past debate on these issues
-
It is not internally driven. Kids learn through RLHF too - but they also learn through RLSF (s=self or subjective). They evaluate/interpret their own performance/knowledge. I don’t think LLMs do that. And, to me, that seems essential to claiming a “mind” - I see the paper Dimitri posted puts a lot more meat on these issues.
I find the “communicative grounding” issue particularly interesting as it gets to the heart of the matter - it seems. LLMs can’t collaboratively engage as they don’t bring their own perspective / intentions/ desires to the table - and that short coming makes it hard, intuitively, to credit them with their own knowledge/ understanding - and hence a mind that could have mental states.
-
@icastico hmm, I feel like the 'communication' route has the same way of dissolving into open issues as the one we are currently on, e.g., how is this example exchange I just ran with Ollama not 'communicatively grounded' in some way?
-
Dimitri Coelho Molloreplied to Ulrike Hahn last edited by
@UlrikeHahn @pbloem As Peter said, the notion of intentionality is tied to things like beliefs and desires, which come with the whole package of agency, etc. In an early paper, I distinguish mental states from cognitive states, and intentionality from aboutness for this reason.
So I think the definition in the OP is baking quite a bit more than it might seem. (Worthwhile to mention the odd fact that philosophers of mind and phil.s of cognitive science are categories that don't overlap much. -
It seems a great example of how the LLM is not collaborating actively - it is simply filling in the blanks using vacuous patterns. That second blurge about why it blurges shows how understanding of the conversation is particularly lacking. Do you think the LLM “fears” anything? Is it really telling you something about itself to guide you to mutual understanding?