“When you pose the question to an LLM, what you are really asking is, “Given the statistical distribution of words in the vast public corpus of text, what are the words most likely to follow the sequence ‘what country is to the south of Rwanda?’” Even if the system responds with the word “Burundi,” this has a different relationship to reality than the human’s answer, and to say the AI “knows” or “believes” Burundi to be south of Rwanda is a category mistake.”
https://www.theverge.com/c/24300623/ai-companions-replika-openai-chatgpt-assistant-romance