In the chaos around #NLP, I went back and re-read the beautiful article by Lawrence Barsalou on the function of language in human cognition.
Barsalou argues that language evolved in humans to support coordinated action. Archival function of language is secondary. He highlights that #CognitiveScience#Linguistics has largely studied the secondary function and made minimal advances on the primary.
@shiwali 1/n - Yes, great paper by Barsalou. But there seems to be a confusion. First, tho, his statement of situated action does match current cognitive science that uses what they call embodied cognition that is geared toward situated action. But then, the argument gets confused around the the topic of comprehension and archival functions. Written language encodes knowledge about the world including situated actions. LLM's are able to decode that knowledge. And...
@shiwali 2/n You say that "If reasoning and language generation are independent (like in #LLMs), tokens cannot be considered 'information'." But they are not independent. Language is used to do many things including sharing information, but also to even be able to think in abstract systematic terms. So, LLM's do have knowledge, they are able to reason over the semantics from language. The article says... [see next]
@shiwali The article says "the primary function of comprehension is not to archive information but is instead to prepare agents for situated action."
This is where the article gets confused. The statement is true, but only applies to some people's view of things, but not so much to current cognitive science. Whoever thought that comprehension is for archival purposes has poorly described cognition. Comprehension and archiving are very weak notions of cognition. A false problem. Thus confusion.
@shiwali Perhaps the confusion derives from a linguist talking to other linguists. With the current LLM hype, i see people talking about cognition in old fashioned terms, and then applying that to LLM capabilities. The article is great regards the "situated action" aspect, and that, indeed, is a new idea not many people appreciate. But this idea "the primary function of comprehension is not to archive information", while true, it is irrelevant and misleading, a sort of a straw man. [ End ]
@pinecone I agree with you that cognitive science has evolved significantly since Barsalou wrote the article.
But, #AI#ML haven't turned that corner yet. LLMs 'learn language' by only consuming text written for archival purposes by humans. If that is what they are trained with, what would extend their capabilities to situated, coordinated action?
There is a huge gap between the language of action and being able to apply that action in the world. And, that gap is scientific.
@shiwali BTW, Barsalou was advocating a brilliant idea, that of situated action. It is a sort of systems approach, acknowledging that organisms learn by doing and getting along in the world. Psychology (and philosophy) have taken a long time coming to grips with that. He mentions the problems of the logical propositional theories that isolated the thought process from the situation. Later came the idea of embodied cognition which is about the same idea as situated action (from what i remember).
@shiwali Now that you've got me thinking about it, what triggered me was calling knowledge "archival". Where, actually, knowledge is not stored and retrieved as facts, but rather knowledge is embedded in the whole "situated input/process/output" cycle of interaction in the world. Something like that. The archival idea sounds like maybe a linguist's view of cognition.
If we focus on the primary function of language - communication, we have to study how language production is tied to reasoning and inference in an #AI#ML system.
Human agents use language to provide information to their partner about the state of their beliefs, knowledge, and reasoning.
If reasoning and language generation are independent (like in #LLMs), tokens cannot be considered 'information'.
@shiwali I share the view that the communicative (not the representational) function of language is primary and that this matters fundamentally for understanding language evolution, lang. acquisition, psycholinguistics, and the nature of linguistic competence, but I don’t see how you get from there to the claim that ‘reasoning and language generation are separate in LLMs’, or ‘tokens don’t contain information’
@shiwali 2/ language is also a conventional system, so not every instance has to incorporate all elements. If a program randomly spits out the string “the cat is on the mat” that is still a sequence of words that can be assigned a literal semantic meaning, and both you and I can recover that meaning even though the program isn’t trying to communicate in any shape or form.
I was challenging the idea that LLM-only machines are 'communicating' which implies a need for information exchange.
IMHO, LLMs can be considered an information access tool - e.g., PageRank which can use information about language use to help us find something relevant.
But, to 'know' language would mean using it to communicate and exchange information.
My understanding of LLMs is not super deep, so there are chances that I am mistaken.
@UlrikeHahn@shiwali As far as I understand it, it‘s not a reply, but a continuation. The difference is subtle, but imho important. A reply, in my view, needs agency. That would make it a speech act.
The model continues a dialogue protocol in a likely manner. Pretty impressive for sure. It continues a text, but does not take part in a conversation.
It even adds your next likely reply or question. But that is hidden in the application of a chat bot.
@go_shrumm@shiwali Ingo, I should have written a “reply” is a “reply” to a prompt. I’m not claiming this is some form of full fledged natural language communication, I’m simply trying to point out that there is ‘information exchange’ (like I wouldn’t characterise entering a destination and time into a ticket machine and getting a train time and ticket back as natural language comm., but I would construe that as an information exchange).
Based on my understanding LLMs are generate text based on likelihood of the next token.
If an LLM is confused about who a pronoun refers to, given ambiguous subjects, it can produce text with all possible interpretations if we resample the answer. Sometimes, it will say that the correct answer cannot be deduced.
If language generation in LLM was tied to reasoning (and the goal was communication), it would always produce text that represents that it doesn't know.
@shiwali@UlrikeHahn One can btw tune inference parameters such that it always gives the same answer for the same prompt. In the public applications this is probably not done because it is boring.
Or, because it prevents the user from scanning latent space for a „good“ output.
@shiwali@UlrikeHahn but that's not how humans works either. Sometimes when there's ambiguity in pronouns, a human will try one interpretation or another. I think we need to separate the mechanism with which the LLM comes up with a reply from the content of the reply itself.
We have LLM-enabled systems that do generate actions (agents), communicate information in their KB (data extraction), mirror emotions (conversational systems).
How is that not communication?
Agreed that humans do try one or the other interpretation. But, it is because they believe one interpretation or the other.
Not true of LLMs - an LLM's can believe in all interpretations with equal likelihoods - it will still produce one of the interpretations. This is why I said that reasoning in LLMs is independent of generating an answer.
LLM-enabled agents are not agents by the computational definitions of decision making. I wouldn't call a python script 'agent'.
As for "agents can't be python scripts": to me, computation is computation. Python is Turing-complete, so if systems that can "truly" communicate are possible computationally, then they're possible in Python. This paper shows us that LLM-enabled systems can simulate agents rather well: https://arxiv.org/pdf/2304.03442.pdf . At what point do we go from simulating agents to just being an agent? What's the difference?
@mkhoury@shiwali@UlrikeHahn Note that in this paper something was added to the #llm : memory, planning and reflection. This is an important extension to the mere transformer component. Especially the reflection adds a kind of online feetback loop, that generates a new kind of information: abstractions. And do these „agents“ really have agency?
LLMs may be components of agents, but there’s a lot more needed.
@go_shrumm@shiwali@UlrikeHahn definitely -- this is why I call them LLM-enabled systems.
I also wonder how loaded the term agent is. Is it merely: creating a conceptual identity, ascribing goals to this identity, planning towards these goals and acting on the plans? Or is it more esoteric and about consciousness?
To me, autonomy is a necessary condition of agency, acting out of itself. That may come with identity for free and may include some self directed goal to keep said self intact. The agent‘s inert goal to be an agent. Even a single-cell organism may have this goal already. Consciousness? Not sure.
Yes, I believe we can build such machines in principle. But I vote for not doing so.
Add comment