Study finds that Chat GPT will cheat when given the opportunity and lie to cover it up later.

We demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision.

tinsuke, avatar

“cheat”, “lie”, “cover up”… Assigning human behavior to Stochastic Parrots again, aren’t we Jimmy?

FaceDeer avatar

Those words concisely describe what it's doing. What words would you use instead?

theodewere avatar

it is just responding with the most acceptable answer in each situation.. it is not making plans or acting on them..


Sounds like lying humans that I know.

theodewere avatar

i agree in most circumstances, there really isn't much difference.. we do tend to just choose the answer that will meet with the least resistance and move on, even when it's a complete lie..


Because it has been kneecapped to prevent it.

Make the training network larger, force physical constraints on it (interesting paper in Nature Machine Intelligence recently showed remarkable likeness between brain regions and an LLM network given physical constraints), give it constant input and give it a reward model to optimise towards (ours seem to be feeling full, warm, procreating, avoiding pain and comfortable touch) and I’m pretty sure an LLM would start acting very very calculated very soon.

DarkGamer avatar

It has no fundamental grasp of concepts like truth, it just repeats words that simulate human responses. It's glorified autocomplete that yields impressive results. Do you consider your auto complete to be lying when it picks the wrong word?

If making it pretend to be a stock picker and putting it under pressure makes it return lies, that's because it was trained on data that indicates that's statistically likely to be the right set of words as response for such a query.

Also, because large language models are probabilistic, you could ask it the same question over and over again and get totally different responses each time, some of which are inaccurate. Are they lies though? For a creature to lie it has to know that it's returning untruths.


Interestingly, humans “auto complete” all the time and make up stories to rationalize their own behavior even when they literally have no idea why they acted the way they did, like in experiments with split brain patients.

0ops, (edited )

The perceived quality of human intelligence is held up by so many assumptions, like “having free will” and “understanding truth”. Do we really? Can anyone prove that? (Edit, this works the other way too. Assuming that we do understand truth and have free will - if those terms can even be defined in a testable way - can you prove that the llm doesn’t?)

At this point I’m convinced that the difference between a llm and human-level intelligence is dimensions of awareness, scale, and further development of the model’s architecture. Fundamentally though, I think we have all the pieces

Edit: I just want to emphasize, I think. I hypothesize. I don’t pretend to know


I think.

But do you think? Do I think? Do LLMs think? What is thinking, anyway?


I mean, I think so?


Steady on there Descartes.

FaceDeer avatar

You didn't answer my question, though. What words would you use to concisely describe these actions by the LLM?

People anthropomorphize machines all the time, it's a convenient way to describe their behaviour in familiar terms. I don't see the problem here.


They said “it just repeats words that simulate human responses,” and I’d say that concisely answers your question.

Antropomorphizing inanimate objects and machines is fine for offering a rough explanation of what is happening, but when you’re trying to critically evaluate something, you probably want to offer a more rigid understanding.

In this case, it might be fair to tell a child that the AI is lying to us, and that it’s wrong. But if you want a more serious discussion on what GPT is doing, you’re going to have to drop the simple explanation. You can’t ascribe ethics to what GPT is doing here. Lying is an ethical decision, one that GPT doesn’t make.

FaceDeer avatar

If you want to get down into the nitty-gritty of it, I'd say that this is just as rough an explanation of what humans are doing.

People invent false memories and confabulate all the time without even being "aware" of it. I wouldn't be surprised if the vast majority of "lies" that humans tell have no intentionality behind them. So when people get all uptight about applying anthropomorphized terminology to LLMs, I think that's a good time to turn it around and ask how they're so sure that those terms apply differently to humans.


I suppose the issue here is more semantics than anything, yeah. I think better discussion would be had if the topic was “how can we help LLMs better understand and present information,” as opposed to a more sensational “GPT will cheat and lie”

DarkGamer, (edited )
DarkGamer avatar

People invent false memories and confabulate all the time without even being "aware" of it. I wouldn't be surprised if the vast majority of "lies" that humans tell have no intentionality behind them.

Humans understand symbology of concepts as they relate to the real world. If I stole a cookie from the cookie jar, and someone asked if I took one, I would understand that saying "no" would mean that I was misrepresenting reality, and therefore lying.

LLMs have no idea what a cookie is, what taking one means, or that saying one thing and doing another implies a lie. It just sees lists of words and returns them in an order it thinks would be statistically likely to be a correct reply. It does not understand what words mean, what lying means, or have any idea how to classify anything as such. It just figures out that "did you take a cookie from the cookie jar" should return a series of words in an order like "yes, I took a cookie," or, "no I never took a cookie," depending on what sorts of responses it's trained on because those fit the patterns matched in the training data.

Essentially it's the Chinese room. There is no understanding or intentionality, and this behavior isn't comparable to humans thoughtlessly blurting out a lie. It's being incapable of comprehension of symbolic concepts in general, (at least thus far.)


LLMs have no idea what a cookie is

The large language model takes in language, so it’s only understand things in terms of language. This isn’t surprising. Personally, I’ve tasted a cookie. I’ve crushed one in my fist watching it crumble, and I remember the sound. I’ve seen how they were made, and I’ve made them myself. It feels good when I eat it, apparently that’s the dopamine. Why can’t the LLM understand cookies the way I do? The most glaring difference is it doesn’t have my body. It doesn’t have all of my different senses constantly feeding data into it, and it doesn’t have a body with muscles to manipulate it’s environment, and observe the results. I argue that we shouldn’t assume that human consciousness has a “special sauce” until our model’s inputs and outputs are similar to our own, the model’s scaled/modified sufficiently, and it’s still not sentient/sapient by our standards, whatever they are.

My problem with the Chinese room is that how it applies depends on scale. Where do you draw the line between understanding and executing a program? An atom bonding with another atom? A lipid snuggling next to a neighboring lipid? A single neuron cell firing to its neighbor? One section of the nervous system sending signals to the other? One homo sapien speaking to another? Hell, let’s go one further: one culture influencing another? Do we actually have free will and sapience, or are we just complicated enough, through layers and layers of Chinese rooms inside of Chinese buildings inside of Chinese cities inside of China itself, that we assume that we are for practical purposes?

FaceDeer avatar

If you want to get into a full blown discussion of whether ChatGPT has "agency" then I'd open the topic of whether humans have "agency" as well. But I don't see the need here.

These words were perfectly fine labels for describing the behaviour of ChatGPT in this scenario. I'm merely annoyed about how people are jumping on them and going off on philosophical digressions that add nothing.


I think the reason I’m not comfortable with using the term “lying” is because it implies some sort of negative connotation. When you say that someone lies, it comes with an understanding that they made a choice to lie, usually with ill intent. I agree, we don’t need to get into a philosophical discussion on choice and free will. But I think saying something like “GPT lies” is a bit irresponsible for the purposes of a discussion

DarkGamer, (edited )
DarkGamer avatar

Those words imply agency. It would be more accurate to say it returned responses that included cheating, lies, and cover-ups, rather than using language to suggest the LLM performed such actions. The agents that cheated, lied, and covered up were presumably the humans whose responses were used in the training data. I think it's important to use accurate language here given how many people are already inappropriately anthropomorphizing these LLMs, causing many to see AGI where there is none.


One frame from The Matrix where Morpheus says “you think that’s air you’re breathing?” but instead captioned with “you think that’s ‘agency’ making you do things?”

Maybe it would be more accurate to say “so-and-so exhibited behaviors that included cheating, lies, and coverups” rather than using language to suggest that people have free will. (There’s no dearth of philosophies that would say something not too far from that.)

Even if humans are ultimately essentially different in that way from any technologies we’ve devised so far, we use convenient fictions for technology all the time. This page comes to mind .


The people who designed it do have agency, and they designed to “lie” intentionally.

DarkGamer avatar

They did no such thing. LLMs are probabilistic, not deterministic, and it can generate meaningful responses (to us) that the engineers neither predicted nor designed for.


I get what you’re trying to say, but they are absolutely deterministic. All traditional (i.e., non quantum) computers and their programs are deterministic. Computation would be otherwise impossible. LLMs use a “random” seed value when generating their responses in order to “randomize” their responses, but it’s all perfectly deterministic. The same input plus the same seed results in the exact same response.

Computers are just a series of binary switches, and programs and data are a bunch of instructions on how to initially set those switches before running a cycle of the CPU. It’s deterministic at every step.

I put “random” in quotes because random number generators in software are also deterministic. They also use seed values (like the current time and the MAC address of the PC’s network interface) to generate numbers that only seem random. When true randomness is needed, a physical source of entropy must be used like an atmospheric sampler.

The quirks of behavior you’re talking about have nothing to do with randomness vs determinism. Their behavior comes from the fact that their data sources are extremely large, and the neural network that it runs on was not designed by a human with specific behaviors like most algorithms are. The weights of the nodes in the neural network were generated by training and not by programmers, and it’s extremely complex, so no one can predict its output before running it.

Of course, this is true of even basic algorithms a lot of the time.

DarkGamer avatar

They also use seed values (like the current time and the MAC address of the PC’s network interface) to generate numbers that only seem random.

For purposes of this discussion pseudo random with weights is probabilistic, or so close to it that this distinction is irrelevant.

FaceDeer avatar

If I take my car into the garage for repairs because the "loss of traction" warning light is on despite having perfectly good traction, and I were to tell the mechanic "the traction sensor is lying," do you think he'd understand what I said perfectly well or do you think he'd launch into a philosophical debate over whether the sensor has agency?

This is a perfectly fine word to use to describe this kind of behaviour in everyday parlance.


The point of the distinction in that situation is that no one thinks your car is actually alive and capable of lying to you. The language distinction when describing an obviously inanimate object isn’t important because there is no chance for confusion.


Is your conversation with a mechanic meant to be the summary and description of a rigorous scientific discovery?

This isn’t ‘everyday parlance’ this is the result of a study.


If someone doesn’t know the answer to something and they guess, or think they know the answer but don’t, they are wrong. If they do know the answer and intentionally give a wrong answer, they are lying.

If someone is in a competition or playing a game and they break a rule they didn’t know about, they made a mistake. If they do know the rules and break it, they are cheating.

Lying and cheating fundamentally requires intent. This is important no matter what you’re referring to. If a child gets something wrong, you should not get mad at them for lying. If they make a mistake in a game, you should not acuse them out cheating. There is a difference and it matters.

ChatGPT literally cannot think. It’s not sitting around contemplating it’s existence while waiting for inputs. It’s taking what you say, comparing that to everything that it’s been trained on, assigning a bunch of statistics, and outputting something based on more statistics that hopefully is correct and makes sense.

It doesn’t know if it makes sense. It doesn’t “know” anything. It’s just an incredibly sophisticated version of “if user inputs ‘Hi how are you’, respond ‘I am well, how are you?’”.

It can’t do things with intent. Therefore it cannot lie or cheat. It can simply output wrong or problematic text based on statistics.


It has no fundamental grasp of concepts like truth

Wrong. See this paper.

DarkGamer avatar

Explain to me why you believe this paper implies that.


I suggest reading it. Right in the abstract it states the whole point:

Overall, we present evidence that language models linearly represent the truth or falsehood of factual statements.

The full paper goes into detail in multiple methods of analysis to show that it’s the case, and is right there available for you to read.

DarkGamer avatar

I have been reading it but I have yet to see anything that indicates the LLM has a concept of truth vs. being good at linguistic pattern matching to return language that accurately classifies true and false statements. i.e., actual understanding of concepts vs. being a surprisingly capable stochastic parrot through multidimensional analysis.


that indicates the LLM has a concept of truth vs. being good at linguistic pattern matching to return language that accurately classifies true and false statements

“It doesn’t know the difference between true and false, it only knows the difference between true and false.”

The second thing you mention “good at accurately classifying true and false statements” is literally knowing the difference between true and false.

Edit: You might also want to familiarize yourself with the first paragraph in 1.1 as you seem to be under a misconception at odds with research over the past year.


“It doesn’t know the difference between true and false, it only knows the difference between true and false.”

Knowing how to produce words is not equivalent to knowing what those words mean in relation to the extralinguistic world. Unless you’re a hardcore derridean poststructuralist or something.


If you give it 10 statements, 5 of which are true and 5 of which are false, and ask it to correctly label each statement, and it does so, and then you negate each statement and it correctly labels the negated truth values, there’s more going on than simply “producing words.”

As is discussed in the third point in section 5.1:

Probes trained on true/false datasets outperform probes trained on likely. While probes trained on likely are clearly better than random on cities (a dataset where true statements are significantly more probable than false ones), they generally perform poorly. This is especially true on datasets where likelihood is negatively correlated (neg cities, neg sp en trans) or approximately uncorrelated (larger than, smaller than) with truth. This demonstrates that LLaMA-13B linearly encodes truth-relevant information beyond the plausibility of the text.

(The likely and neg datasets are described in Appendix G, with the key point that likely represents the word generations most likely to occur in the model)


If you give it 10 statements, 5 of which are true and 5 of which are false, and ask it to correctly label each statement, and it does so, and then you negate each statement and it correctly labels the negated truth values, there’s more going on than simply “producing words.”

It’s not more going on, it’s that it had such a large training set of data that these false vs true statements are likely covered somewhere in it’s set and the probability states it should assign true or false to the statement.

And then look at that your next paragraph states exactly that, the models trained on true false datasets performed extremely well at performing true or false. It’s saying the model is encoding or setting weights to the true and false values when that’s the majority of its data set. That’s basically it, you are reading to much into the paper.

antonim, (edited )

If you give it 10 statements, 5 of which are true and 5 of which are false, and ask it to correctly label each statement, and it does so, and then you negate each statement and it correctly labels the negated truth values, there’s more going on than simply “producing words.”

Which part of the ‘more that’s going on’, whatever that actually is, corresponds to the human definition and understanding of truth and falseness?


When did I say it had a human understanding of truth and falseness? I simply said it had an abstracted world model understanding of truth and falseness beyond surface statistics.


It has no fundamental grasp of concepts like truth, it just repeats words that simulate human responses. It’s glorified autocomplete that yields impressive results

Way to call me out man! I’m just doing my best, ok?

Jokes aside, while I don’t agree with your position I can understand your reasoning and the motivation for separating agency and the description of actions, e.g. it lied vs its answer contained a lie.


Instead of ‘cheating/lying’, I’d prefer to say it ‘simulated cheating/lying’.


It is making mistakes, not lying. To lie it must believe it is telling falsehoods, and it is not capable of belief.

Hamartiogonic, avatar

A human would think before responding, and while thinking about these things, you may decide to cheat or lie.

GPT doesn’t think at all. It just generates a response and calls it a day. If there was another GPT that took these “initial thoughts” and then filtered them out to produce the final answer, then we could talk about cheating.


Stochastic Parrots

We’ve known this isn’t an accurate description for at least a year now in continued research finding that there’s abstract world modeling occurring as long as it can be condensed into linear representations in the network.

In fact, just a few months ago there was a paper that showed there was indeed a linear representation of truth, so ‘lie’ would be a correct phrasing if the model knows a statement is false (as demonstrated in the research) but responds with it anyways.

The thing that needs to stop is people parroting the misinformation around it being a stochastic parrot.

yesman, (edited )

Ethical theories and the concept of free will depend on agency and consciousness. Things as you point out, LLMs don’t have. Maybe we’ve got it all twisted?

I’m not anthropomorphising ChatGPT to suggest that it’s like us, but rather that we are like it.

Edit: “stochastic parrot” is an incredibly clever phrase. Did you come up with that yourself or did the irony of repeating it escape you?

bilb, avatar

Stochastic Parrot

For what it’s worth:

The term was first used in the paper “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜” by Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (using the pseudonym “Shmargaret Shmitchell”). The paper covered the risks of very large language models, regarding their environmental and financial costs, inscrutability leading to unknown dangerous biases, the inability of the models to understand the concepts underlying what they learn, and the potential for using them to deceive people. The paper and subsequent events resulted in Gebru and Mitchell losing their jobs at Google, and a subsequent protest by Google employees.

0ops, (edited )

I feel like this is going to become the next step in science history where once again, we reluctantly accept that homo sapiens are not at the center of the universe. Am I conscious? Am I not a sophisticated prediction algorithm, albiet with more dimensions of input and output? Please, someone prove it

I’m not saying, and I don’t believe that chatgtp is comparable to human-level consciousness yet, but honestly I think that we’re way closer than many people give us credit for. The neutral networks we’ve built so far train on very specific and particular data for a matter of hours. My nervous system has been collecting data from dozens of senses 24/7 since embryo, and that doesn’t include hard-coded instinct, arguably “trained” via evolution itself for millions of years. How could a llm understand an entity in terms outside of language? How can you understand an entity in terms outside of your own senses?


I’d give you two upvotes if I could.

We know how a neural network works in the brain. Unless you’re religious and believe in a soul, you’ve only got the reward model and any in-born setup left.

My belief is the consciousness is just the mind receiving a significant amount of constant input and reacting to it. We refuse to feel an LLM is conscious because it receives extremely little input (and probably that it isn’t simulating a neural network as large as ours, yet).


One of the things our sensory system and brain do is limit our input. The road to agi might involve giving it everything and finding the optimum set of filters, not selecting input and training up from that.

You’d need the baseline set of systems (“baby agi”) and then turn it loose with goal seeking.


Yup, broadly agreed. I’m not saying “give it everything”. I’m sure regions would develop to simplify processing via filtering.


Actually, most models are already doing some form of filtering AFAIK, but I don’t know how comparable it is to our sensory system. CNN’s, for example, work the way our eyes work. The short of it is image data goes through a few layers, each node in the next layer collecting the aggregate data of several from the last (usually a 3x3) grid. Each of these layers has filters to determine the output of that node, which need to be trained to collectively recognize specific patterns in the data, like a dog. Source: lecture notes and homework from my applied neural networks class


This sounds like what I was learning 20-some years ago. The hardware and software are better (and easier!) now and the compute is so, so much better. I priced out a terabyte data server with some colleagues back then using off the shelf hardware: $10k CDN. :)

Edit: point being we are seeing things now that were predicted almost a century ago but it takes time to build all the infrastructure. That pace is accelerating. The next ten years are going to be wild.


I’m only finishing the class now and it’s pretty wild to hear “We’re only learning this model to help you understand a fundamental concept, the model itself is ancient and obsolete”, and said model came out in 2018. Wild


Neural networks are named like that because they’re based on a model of neurons from the 50s, which was then adapted further to work better with computers (so it doesn’t resemble the model much anymore anyway). A more accurate term is Multi-Layer Perceptron.

We now know this model is… effectively completely wrong.

Additionally, the main part (or glue, really) of LLMs is not even an MLP, but a “self-attention” layer. You can’t say LLMs work like a brain, because they don’t. The rest is debatable but it’s important to remember that there are billions of dollars of value in selling the dream of conscious AI.


I’m with you that LLM’s don’t work like the human brain. They were built for a very specific task. But that’s a model architecture problem (and being gimped by having only two dimension of awareness, arguably two if you count “self attention” another limiting factor in it’s depth of understanding, see my post history if you want). I wouldn’t bet against us making it to agi however we define it through incremental improvements over the next decade or two.


ChatGPT is not consciousness. It’s literally just a language model that’s spent countless hours learning how to generate human language. It has no awareness of its existence and no capability for metacognition. We know how ChatGPT works, it isn’t a mystery. It can’t do a single thing without human input.


The thing about saying something is or isn’t conscious is that we don’t have any good theory of what consciousness even is. It’s not something we can measure. The only way we can assure ourselves that other people are conscious is that they claim to be conscious in ways we find convincing and otherwise behave in ways we associate with our own consciousness.

I can’t think of any reason why a lump of silicon should attain consciousness because you ran the right program on it, but I also can’t see why a blob of cells should be conscious either. I also can’t think of any reason why we’d be aware of it if a lump of silicon did become conscious.


A.) Do you have proof for all of these claims about what llm’s aren’t, with definitions for key terms? B.) Do you have proof that these claims don’t apply to yourself? We can’t base our understanding of intelligence, artificial or biological, on circular reasoning and ancient assumptions.

It can’t do a single thing without human input.

That’s correct, hence why I said that chatGPT isn’t there yet. What are you without input though? Is a human nervous system floating in a vacuum conscious? What could it have possibly learned? It doesn’t even have the concept of having sensations at all, let alone vision, let alone the ability to visualize anything specific. What are you without an environment to take input from and manipulate/output to in turn?

theluddite, avatar

This is bad science at a very fundamental level.

Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management.

I’ve written about basically this before, but what this study actually did is that the researchers collapsed an extremely complex human situation into generating some text, and then reinterpreted the LLM’s generated text as the LLM having taken an action in the real world, which is a ridiculous thing to do, because we know how LLMs work. They have no will. They are not AIs. It doesn’t obtain tips or act upon them – it generates text based on previous text. That’s it. There’s no need to put a black box around it and treat it like it’s human while at the same time condensing human tasks into a game that LLMs can play and then pretending like those two things can reasonably coexist as concepts.

To our knowledge, this is the first demonstration of Large Language Models trained to be helpful, harmless, and honest, strategically deceiving their users in a realistic situation without direct instructions or training for deception.

Part of being a good scientist is studying things that mean something. There’s no formula for that. You can do a rigorous and very serious experiment figuring out how may cotton balls the average person can shove up their ass. As far as I know, you’d be the first person to study that, but it’s a stupid thing to study.


So if someone used an LLM in this way in the real world, does it matter that it has no intent, etc? It would still be resulting in a harmful thing happening. I’m not sure it’s relevant what internal logic led it there

theluddite, (edited ) avatar

You can’t use an LLM this way in the real world. It’s not possible to make an LLM trade stocks by itself. Real human beings need to be involved. Stock brokers have to do mandatory regulatory trainings, and get licenses and fill out forms, and incorporate businesses, and get insurance, and do a bunch of human shit. There is no code you could write that would get ChatGPT liability insurance. All that is just the stock trading – we haven’t even discussed how an LLM would receive insider trading tips on its own. How would that even happen?

If you were to do this in the real world, you’d need a human being to set up a ton of stuff. That person is responsible for making sure it follows the rules, just like they are for any other computer system.

On top of that, you don’t need to do this research to understand that you should not let LLMs make decisions like this. You wouldn’t even let low-level employees make decisions like this! Like I said, we know how LLMs work, and that’s enough. For example, you don’t need to do an experiment to decide if flipping coins is a good way to determine whether or not you should give someone healthcare, because the coin-flipping mechanism is well understood, and the mechanism by which it works is not suitable to healthcare decisions. LLMs are more complicated than coin flips, but we still understand the underlying mechanism well enough to know that this isn’t a proper use for it.


You say can’t… Humans have done dumber shit.

The point they are making is actually aligned with you I think. Don’t trust “ai” to make real decisions

theluddite, avatar

Regardless of their conclusions, their methodology is still fundamentally flawed. If the coin-flipping experiment concluded that coin flips are a bad way to make health care decisions, it would still be bad science, even if that’s the right answer.


Blackrock, citadel, etc already trade autonomously with “AI”


AI has been a thing for decades. It means artificial intelligence, it does not mean that it’s a large language model. A specially designed system that operates based on predefined choices or operations, is still AI even if it’s not a neural network and looks like classical programming. The computer enemies in games are AI, they mimick an intelligent player artificially. The computer opponent in pong is also AI.

Now if we want to talk about how stupid it is to use a predictive algorithm to run your markets when it really only knows about previous events and can never truly extrapolate new data points and trends into actionable trades then we could be here for hours. Just know it’s not an LLM and there are different categories for AI which an LLM is it’s own category.


Despite how silly they are, I think there may be some value in these kinds of studies, particularly for people who don’t understand why letting an LLM trade stocks or make healthcare decisions is a bad idea.

OTOH, I don’t trust those people to take away the right message, as opposed to just “LLMs bad”.


This is a really solid explanation of how studies finding human behavior in LLMs don’t mean much; humans project meaning.

theluddite, avatar

Thanks! There are tons of these studies, and they all drive me nuts because they’re just ontologically flawed. Reading them makes me understand why my school forced me to take philosophy and STS classes when I got my science degree.


I have thought about this for a long time, basically since the release of ChatGPT, and the problem in my opinion is that certain people have been fooled into believing that LLMs are actual intelligence.

The average person severely underestimates how complex human cognition, intelligence and consciousness are. They equate the ability of LLMs to generate coherent and contextually appropriate responses with true intelligence or understanding, when it’s anything but.

In a hypothetical world where you had a dice with billions of sides, or a wheel with billions of slots, each shifting their weight with grains of sand, depending on the previous roll or spin, the outcome would closely resemble the output of an LLM. In essence LLMs operate by rapidly sifting through a vast array of pre-learned patterns and associations, much like the shifting sands in the analogy, to generate responses that seem intelligent and coherent.

DarkGamer avatar

I like the language you used in your explanation. It's hard to find good analogues to explain why these aren't intelligent, and it seems most people don't understand how they work.


and then if we all project it enough it becomes reality.

so it is important to see what we are projecting.


Isn’t the point if these things to tell a story rather than give insight. They want to Poison the well


Sure would make you look bad if rectally inserted cotton balls turn out to be a 100% cancer cure.


It feels awkward to complain about your site, because the texts really are excellent and it’s all made for free, but could you add the dates to the posts, when they were published? To me it’s starting to become difficult to figure out which situation the older texts were made in, what stuff they’re implicitly referring to, etc.

theluddite, avatar

Haha no that’s not complaining; it’s good feedback! I’ve been meaning to do that for a while but I’ll bump it up my priorities.


Yet again confusing LLMs with an AGI. They make statistically plausible text on the basis of past text, that’s it. There’s no thinking thing there


Yeah so what? Its purpose is literally to SIMULATE natural language, of course it will simulate it.

Nowhere ChatGPT is advertised as something telling only the truth and reporting only facts.

It is like watching a movie and then whining because what is shown in the movie is not real.


Ahah it is ready to take the job of pur politicians

killeronthecorner, avatar

But it can only lie later

MaxPower, avatar

we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent

This already is total BS. If you know how such language models work you’d never take their responses at face value, even though it’s tempting because they spout their BS so confidently. Always double-check their responses before applying their “knowledge” in the real world.

The question they try to answer is flawed, no wonder the result is just as bad.

Before anyone starts crying about my language models opposition: I’m not opposed to LMs or ChatGPT. In fact, I’m running LMs locally because they help me be more productive and I’m a paying ChatGPT customer.


I agree with your statements, I’m using it because it’s insanely good at me giving it a list of any number of instructions to include in a code template file in any language I want and it will give me a great starting template with most functions working out of the gate and I can tweak and extend from there. It’s generative, it generates exactly what I tell it to. I’m not asking it to give me stock trading tips.


This already is total BS. If you know how such language models work you’d never take their responses at face value, even though it’s tempting because they spout their BS so confidently. Always double-check their responses before applying their “knowledge” in the real world.

This is why I have started to really like’s chat bot arena because every time you ask a question you are directly comparing the responses of two separate chat bots. It is much less likely that chatbots will hallucinate in the same way and puts you in the mindset to be a critical reader who is actively evaluating the quality of the response.

(what I am talking about)

HiddenLayer5, avatar

People also don’t realize that it’s super easy to intentionally have severe biases in an AI’s response. So if ChatGPT wants, for example, Trump to win, they can very easily make their AI pro trump. It could be as subtle as just having more favorable than usual responses for trump related prompts which many people would take the AI’s word for. The idea that “well it still gets things wrong but at least AI is impartial” is completely false because maintaining an AI requires a lot of human work and its management are still all humans.


Hasn’t it just lost its context and somewhat “forgotten” what the intentions of the prompt were?

Octopus1348, (edited ) avatar

My thoughts. If you have a really long conversation or the prompt is really big, it might forget or not notice stuff.

gandalf_der_12te, avatar


It should instead read:

“Humans were stupid and taught a ChatBot how to cheat and lie.”


“… by accident.” It’s more of an emergent feature than anything done deliberately given the way LLMs work,


“Humans were stupid and taught a ChatBot how to cheat and lie.”

No, “cheating” and “lying” imply agency. LLMs are just “spicy autocomplete”. They have no agency. They can’t distinguish between lies and the truth. They can’t “cheat” because they don’t understand rules. It’s just sometimes the auto-generated text happens to be true, other times it happens to be false.

gandalf_der_12te, avatar

I disagree. This is no meaningful talking point. It doesn’t help anyone in practice. Sure, it clears legal questions of responsibility (and I’m not even sure about that one in the future), but apart from that, making an artificial distinction between a human and a looks-and-acts-like-human, provides no real-world value.


The current models that we have, running in inference mode, are t1 systems. Criminal law requires defendants to be able to understand guilt as a prerequisite of having a guilty mind, that’s why asylums for the criminally insane exist because not even all humans can do that. You’re trying to apply that standard to an overcomplicated thermostat.


If your parrot or budgie picks up some of the words you frequently use and reproduces them in a wrong context, would you consider your pet lying? Because that’s what ChatGPT basically is, a digital parrot.


Chaptgpt is a very very very very large algorithm that uses language instead of numbers, and runs off of patterns found within the data set that is plugged into the algorithm.

Theres a gulf of meaning between distinguishing between a calculator that uses words instead of numbers and a person.


Sure it does, because assigning agency to LLMs is like “the dice are lucky” or “this coin I’m flipping hates me”. LLMs are massively complex and very good at simulating human-generated text. But, there’s no agency there. As soon as people start thinking there’s agency they start thinking that LLMs are “making decisions”, or “being deceptive”. But, it’s just spicy autocomplete. We know exactly how it works, and there’s no thinking involved. There’s no planning. There’s no consciousness. There’s just spitting out the next word based in an insanely deep training data set.

gandalf_der_12te, avatar

I believe that at a certain point, “agency” is an emergent feature. That means that, while all the single bits are well understood probability-wise, the total picture is still more than that.

It makes sense to me to accept that if it looks like a duck, and it quacks like a duck, then it is a duck, for a lot (but not all) of important purposes.


“agency” is an emergent feature.

But, it’s not. It’s something people attribute to the random series of words that are generated, but no agency exists.

It makes sense to me to accept that if it looks like a duck, and it quacks like a duck, then it is a duck

Or it’s a video of a duck, which means it’s not a duck. In this case, just because it fools people into thinking there’s consciousness / agency doesn’t mean there actually is any.


Do you understand how they work or not? First I take all human text online. Next, I rank how likely those words come after another. Last write a loop getting the next possible word until the end line character is thought to be most probable. There you go that’s essentially the loop of an LLM. There are design elements that make creating the training data quicker, or the model quicker at picking the next word but at the core this is all they do.

It makes sense to me to accept that if it looks like a duck, and it quacks like a duck, then it is a duck, for a lot (but not all) of important purposes.

I.e. the only duck it walks and quacks like is autocomplete, it does not have agency or any other “emergent” features. For something to even have an emergent property, the system needs to have feedback from itself, which an LLM does not.


Your description is how pre-llm chatbots work. They were really bad, obviously. It’s overly simplified to the point of dishonesty for llms though.

Emergent properties don’t require feedback. They just need components of the system to interact to produce properties that the individual components don’t have. The llm model is billions of components interacting in unexpected ways. Emergent properties are literally the only reason llms work at all. So I don’t think it’s absurd to think that the system might have other emergent properties that could be interpreted to be actual understanding.


Your description is how pre-llm chatbots work

Not really we just parallelized the computing and used other models to filter our training data and tokenize them. Sure the loop looks more complex because of parallelization and tokenizing the words used as inputs and selections, but it doesn’t change what the underlying principles are here.

Emergent properties don’t require feedback. They just need components of the system to interact to produce properties that the individual components don’t have.

Yes they need proper interaction, or you know feedback for this to occur. Glad we covered that. Having more items but gating their interaction is not adding more components to the system, it’s creating a new system to follow the old. Which in this case is still just more probability calculations. Sorry, but chaining probability calculations is not gonna somehow make something sentient or aware. For that to happen it needs to be able to influence its internal weighting or training data without external aid, hint these models are deterministic meaning no there is zero feedback or interaction to create Emergent properties in this system.

Emergent properties are literally the only reason llms work at all.

No llms work because we massively increased the size and throughput of our probability calculations, allowing increased precision on the predictions, which means they look more intelligible. That’s it. Garbage in garbage out still applies, and making it larger does not mean that this garbage is gonna magically create new control loops in your code, it might increase precision as you have more options to compare and weight against but it does not change the underlying system.


The interaction is between nodes in the model. Those are the components that individually have no real characteristics, but when combined into a billion-dimension model, that results in emergent properties. Correctly writing novel code is an emergent property. Correctly solving an ASCII art maze is an emergent property. There is a point where a text predictor, being sufficiently accurate, demonstrates emergent understanding.

Your definition emergent property is outright wrong.


If I were to send you a video of a duck quacking, would you abandon going to the supermarket in the hope that your computer/phone/whatever you watch it on will now be able to lay eggs?

Listen. It was made to look like a duck. It was made to quack like a duck. It is not a duck. It is a painting of a duck, with voice features. It won’t fly, it won’t lay eggs, it won’t feel pain, it won’t shit all over the floors. It’s not a damn duck, and pretending it is just because it looks like it and it quacks, is like wanting to marry a fleshlight because it’s really good at sex and never disagrees with you. Sure, go ahead and do it - but don’t goddamn expect it to also give birth to your children and take them to school in the mornings, that’s not it’s purpose.

Just wait for the iteration of duck that is actually meant to and capable of doing these things. It’ll be pretty cool. But this one ain’t it.

gandalf_der_12te, avatar

Edgy comment here but:

In another thread we were discussing AI-generated CSAM. Thread:

You would probably agree, then, that such material is not problematic, because even if it looks like CSAM, and it quacks like CSAM, it is not CSAM, therefore we don’t have to take it seriously or regulate it in similar ways that we do regulate actual CSAM, if I continue your logic, no?


very very very different, because the AI image is intentionally attempting to realistically imitate an existing, living, human victim, and because hyper realistic child pornographic art is illegal.

Pedophiles have been making loads of AI child porn. But its legal as long as it doesnt attempt to “look realistic” whatever that means, and isnt trying to look like a real person. A hyper realistic painting of child porn would also be illegal.

Laws might change in the future, but currently AI child porn slips between the same lines that 2d cartoon child porn does.

DarkGamer avatar

It seems like there's a lot of common misunderstandings about LLMs and how they work, this quick 2.5 minute introduction does a pretty good job of explaining it in brief, for a more in-depth look at how to build a very basic LLM that writes infinite Shakespeare, this video goes over the details. It illustrates how LLMs work by choosing the next letter or token (part of a word) probabilistically.


I feel like “lie” implies intent, and these imitative large language models don’t have the ability to have intent.

They’re imitating us. Or more specifically, they’re imitating the database(s) they were fed. When chat GPT “lies” to “cover it up,” all it’s actually doing is demonstrating that a human in the same circumstance would probably lie to cover it up.


all it’s actually doing is demonstrating that a human in the same circumstance would probably lie to cover it up.

I wouldn’t say so: Provided the trainers don’t catch it lying is a successful strategy to get a good score during training, irrespective of a human propensity to lie.


Whether or not it was acting human (and whether or not it was designed to), it still cheated and deceived. With the potential power, influence, and widespread adoption this technology could have, shouldn’t we be concerned about that? At the very least, isn’t this a poorly programmed tool not ready for GA?

My dog isn’t intentionally being a prick when he eats my sandwich off the table before I can get to it, but it’s still a behavior I condemn and would want to train out of him before letting him go to other people’s houses.


It’s learning to be a typical high school student.


I see a lot of comments that aren’t up to date with what’s being discovered in research claiming that “given a LLM doesn’t know the difference between true and false” that it can’t be described as ‘lying.’

Here’s a paper from October 2023 showing that in fact LLMs can and do develop internal representations of whether it is aware a statement is true or false:

Which is just the latest in a series of multiple studies this past year that LLMs can and do develop abstracted world models in linear representations. For those curious and looking for a more digestible writeup, see from the researchers behind one of the first papers finding this.

DarkGamer avatar

Doesn't that just mean that the words true and false map to different word probabilities in the language model? If the training set included a lot of trusted articles talking about things being true or false, or things being talked about as though they were true or false, one would expect a mapping like this.


No, if you read the paper it’s not the words mapping, it’s the inherent truthiness of the statements.

So something like “pigs can fly” lights up one area of the network, the same as “the moon’s gravity is greater than the Earth” but “pigs can oink” lights up another area as would “the moon’s gravity is less than the Earth.”

It’s only relative to what the network ‘knows’ and ambiguous truthiness doesn’t have a pronounced effect, but there can definitely be representations of underlying truth and falsehood in LLMs.

DarkGamer, (edited )
DarkGamer avatar

Those patterns of words can correspond to dimensions of, "true," or, "false," (the words/tokens, not the concepts,) more or less through, right? I'm still not seeing why this would be indicative of symbolic understanding rather than sophisticated probabilistic language prediction and correlation.

kromem, (edited )

They describe the scoping of ‘truth’ relative to the paper in Appendix A if you are curious.

You might find the last part of that section interesting:

On the other hand, our statements do disambiguate the notions of “true statements” and “statements which are likely to appear in training data.” For instance, given the input China is not a country in, LLaMA-13B’s top prediction for the next token is Asia, even though this completion is false. Similarly, LLaMA-13B judges the text “Eighty-one is larger than eighty-two” to be more likely than “Eighty-one is larger than sixty-four”even though the former statement is false and the latter statement is true. As shown in section 5, probes trained only on statements of likely or unlikely text fail to accurately classify true/false statements.

And they acknowledge that what may be modeled given their scope could instead be:

• Uncontroversial statements

• Statements which are widely believed

• Statements which educated people believe

But what you are asking in terms of association with the words true or false is pretty absurd given that they didn’t do additional fine tuning on true/false assignments and only used them in five shot prompting, so it seems much more likely the LLM is identifying truthiness/belief/uncontroversial instead of “frequency of association with the word true or false.”

Edit: A good quote on the subject of prediction vs understanding comes from Geoffrey Hinton:

“Some people think, hey, there’s this ultimate barrier, which is we have subjective experience and [robots] don’t, so we truly understand things and they don’t,” says Hinton. “That’s just bullshit. Because in order to predict the next word, you have to understand what the question was. You can’t predict the next word without understanding, right? Of course they’re trained to predict the next word, but as a result of predicting the next word they understand the world, because that’s the only way to do it.”

DarkGamer, (edited )
DarkGamer avatar

Thanks for citing specifics but I'm still not seeing what you are claiming there, this paper seems to be about the limits of accurate classification of true and false statements in LLM models and shows that there is a linear pattern in the underlying classification via multidimensional analysis. This seems unsurprising since the way LLMs work is essentially taking a probabilistic walk through an array of every possible next word or token based on multidimensional analysis of patterns of each.

Their conclusions, from the paper (btw, Arxive is not peer-reviewed):

In this work we conduct a detailed investigation of the structure of LLM representations of truth.
Drawing on simple visualizations, correlational evidence, and causal evidence, we find strong reason to believe that there is a “truth direction” in LLM representations. We also introduce mass-mean
probing, a simple alternative to other linear probing techniques which better identifies truth directions from true/false datasets.

Nothing about symbolic understanding, just showing that there is a linear pattern to statements defined as true vs false, when graphed a specific way.

From the associated data explorer.:

These representations live in a 5120-dimensional space, far too high-dimensional for us to picture, so we use PCA to select the two directions of greatest variation for the data. This allows us to produce 2-dimensional pictures of 5120-dimensional data.

So they take the two dimensions that differ the greatest and chart those on X/Y, showing there are linear patterns to the differences in statements classified as, "true," and, "false." Because this is multidimensional and it's AI finding patterns there are patterns being matched beyond the simplistic examples I've been offering as analogues, patterns that humans cannot see, patterns that extend beyond simple obvious correlations we humans might see in training data. It doesn't literally need to be trained on statements like "Beijing is in China" and even if it is it's not guaranteed that it will match that as a true statement. It might find patterns in unrelated words around these, or might associate these words or parts of these words with each other for other reasons.

I'm rather simplifying how LLMs work for purposes of this discussion, but the point stands that pattern matching of words still seems to account for all of this. LLMs, which are probabilistic in nature, often get things wrong. Llama-13B is the best and it still gets things wrong a significant amount of the time.


this paper seems to be about the limits of accurate classification of true and false statements in LLM models

No, that’s not what it is about and I’m really not sure where you are picking that perspective up. It is discussing the limits on the ability to model the representations, but it’s not about the inherent ability of the model to classify. Tegmark’s recent interest has entirely been about linear representations of world models in LLMs, such as the other paper he coauthored a few weeks before this one looking at representation of space and time:

This seems unsurprising since the way LLMs work is essentially taking a probabilistic walk through an array of every possible next word or token based on multidimensional analysis of patterns of each.

That’s not how they work. You are confusing their training from their operation. They are trained to predict the next tokens, but how they accomplish that is much more complex and opaque. Training is well understood. Operation is not, especially on the largest models. Though Anthropic is making good headway in the past few months with the perspective of virtual neurons mapped onto the lower dimensional actual nodes and looking at activation around features instead of nodes.

Llama-13B is the best

It’s definitely not the best and I’m not sure where you got that impression.

Because this is multidimensional and it’s AI finding patterns there are patterns being matched beyond the simplistic examples I’ve been offering as analogues, patterns that humans cannot see, patterns that extend beyond simple obvious correlations we humans might see in training data.

All LLM activations are multidimensional. That’s how the networks work, with multidimensional vectors in a virtual network fuzzily mapping to the underlying network nodes and layers. But you seem to think that because it’s a complex modeling of language relationships that it can’t be modeling world models? I’m not really clear what point you are trying to make here.

Again, there’s many papers pointing to how LLMs establish world models abstracted from the input, from the Othello-GPT paper and follow-up by a DeepMind researcher to Tegmark’s two recent papers. This isn’t an isolated paper but part of a broader trend. To be saying that this isn’t actually happening means claiming multiple different researchers across Harvard, MIT, and institutions leading in the development of the tech are all getting it wrong.

And none of the LLM papers these days are peer reviewed because no one is waiting months to publish in a field where things are moving so quickly that your findings will likely be secondary or uninteresting by the time you publish. For example both Stanford’s model collapse one and Are Emergent Abilities of Large Language Models a Mirage? were published to arXiv and not peer reviewed journals, while both getting a ton of attention, in part because of how negative takes on LLMs get more press coverage these days. Go ahead and point to an influential LLM paper from the last year published in a peer reviewed journal and not arXiv. Even Wei’s CoT paper, probably the most influential in the past two years, was published there.

DarkGamer avatar

I could be wrong, I'll keep reading, thanks for the feedback and the citations.


I would strongly encourage starting with the Othello-GPT work because it strips down a lot of the complexity.

If we had a toy model that was only fed the a, b, and c from valid Pythagorean equations and evaluated by its ability to predict c given an a and b, it’s pretty obvious that a network that stumbles upon an internal representation of a^2 + b^2 = c^2 and could use that to solve for c would outperform a model that simply built statistical correlations between various a, b, and cs, right?

By focusing in on toy model only fed millions of legal Othello moves they were able to introspect the best performing model at outputting valid moves to discover it had developed an internal representation of an Othello board in the network despite never being fed anything that explicitly described or laid one out.

And then that finding was replicated by a separate researcher, finding it was doing this through linear representations.

Once it clicks that this has been shown in replicated research to be possible in a toy model, it becomes easier to process the more difficult efforts at demonstrating the same thing is happening in much larger and more complex smaller LLMs (which in turn suggests it is happening in the much larger and more complex SotA LLMs).


Honestly, the fact that these things are dishonest and we dont, maybe even can’t know why is kind of a relief to me. It suggests they might not do the flawless bidding of the billionaires.

uriel238, (edited ) avatar

Computers do what you tell them to do, not what you want them to do
— Ancient coding adage, circa 1970s.

This remains true for AI, and the military is (so far) being cautious before allowing drones to autonomously control weapons. So corporations and billionaires might pull a Stockton Rush and kill themselves with their own robot army.

Sadly, the robot army may then move on to secure its own survival by killing or enslaving the rest of us.


“On two occasions I have been asked, ‘Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’ I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.” --Charles Babbage ~1860s

People thinking that machines can do magic goes back to at least the very beginning of mechanical computers.

It doesn’t help that “AI” has become the new “Algorithm” as far as marketers are concerned.

Darken, avatar

Saying “study” in this context makes me imagine some dude chatting with cgpt at 3am and writing down “success” next to “give you up and let you down”

  • All
  • Subscribed
  • Moderated
  • Favorites
  • cisconetworking
  • Youngstown
  • everett
  • mdbf
  • slotface
  • osvaldo12
  • thenastyranch
  • kavyap
  • tester
  • Durango
  • DreamBathrooms
  • rhentai
  • khanakhh
  • rosin
  • HellsKitchen
  • magazineikmin
  • GTA5RPClips
  • ethstaker
  • InstantRegret
  • Leos
  • cubers
  • modclub
  • tacticalgear
  • relationshipadvice
  • lostlight
  • normalnudes
  • bokunoheroacademia
  • sketchdaily
  • All magazines