Results of the "Can you tell which images are AI generated?" survey

Previous posts: programming.dev/post/3974121 and programming.dev/post/3974080

Original survey link: forms.gle/7Bu3Tyi5fufmY8Vc8

Thanks for all the answers, here are the results for the survey in case you were wondering how you did!

Edit: People working in CS or a related field have a 9.59 avg score while the people that aren’t have a 9.61 avg.

People that have used AI image generators before got a 9.70 avg, while people that haven’t have a 9.39 avg score.

Edit 2: The data has slightly changed! Over 1,000 people have submitted results since posting this image, check the dataset to see live results. Be aware that many people saw the image and comments before submitting, so they’ve gotten spoiled on some results, which may be leading to a higher average recently: docs.google.com/…/1MkuZG2MiGj-77PGkuCAM3Btb1_Lb4T…

mykl,
@mykl@lemmy.world avatar

Oof. I got about 65% on the images I hadn’t seen in the post. I must be pretty close to being replaceable by an adversarial network.

gveltaine,

That avocado and tomato post took me out, that and the Legos. Very impressive.

The most obvious ai one for me was the last cat picture, somehow it just screamed ai

Custoslibera,

What I learnt is that I’m bad at that task.

squirrelwithnut,

Sketches are especially hard to tell apart because even humans put in extra lines and add embellishments here and there. I’m not surprised more than 70% of participants weren’t able to tell that one was generated.

bitsplease,

One thing I’d be interested in is getting a self assessment from each person regarding how good they believe themselves to have been at picking out the fakes.

I already see online comments constantly claiming that they can “totally tell” when an image is AI or a comment was chatGPT, but I suspect that confirmation bias plays a big part than most people suspect in how much they trust a source (the classic “if I agree with it, it’s true, if I don’t, then it’s a bot/shill/idiot”)

ILikeBoobies,

With the majority being in CS fields and having used ai image generation before they likely would be better at picking out than the average person

bitsplease,

You’d think, but according to OP they were basically the same, slightly worse actually, which is interesting

ILikeBoobies,

The ones using image generation did slightly better

I was more commenting it to point out that it’s not necessary to find that person who can totally tell because they can’t

lloram239,

Even when you know what you are looking for, you are basically pixel hunting for artifacts or other signs that show it’s AI without the image actually looking fake, e.g. the avocado one was easy to tell, as ever since DALLE1 avocado related things have been used as test images, the thispersondoesnotexist.com one was obvious due to how it was framed and some of the landscapes had that noise-vegetation-look that AI images tend to have. But none of the images look fake just by themselves, if you didn’t specifically look for AI artifacts, it would be impossible to tell the difference or even notice that there is anything wrong with the image to begin with.

Spzi,

Right? A self-assessed skill which is never tested is a funny thing anyways. It boils down to “I believe I’m good at it because I believe my belief is correct”. Which in itself is shady, but then there are also incentives that people rather believe to be good, and those who don’t probably rather don’t speak up that much. Personally, I believe people lack the competence to make statements like these with any significant meaning.

doggle,

Having used stable diffusion quite a bit, I suspect the data set here is using only the most difficult to distinguish photos. Most results are nowhere near as convincing as these. Notice the lack of hands. Still, this establishes that AI is capable of creating art that most people can’t tell apart from human made art, albeit with some trial and error and a lot of duds.

blueberrypie,

These images were fun, but we can’t draw any conclusions from it. They were clearly chosen to be hard to distinguish. It’s like picking 20 images of androgynous looking people and then asking everyone to identify them as women or men. The fact that success rate will be near 50% says nothing about the general skill of identifying gender.

bitsplease,

Idk if I’d agree that cherry picking images has any negative impact on the validity of the results - when people are creating an AI generated image, particularly if they intend to deceive, they’ll keep generating images until they get one that’s convincing

At least when I use SD, I generally generate 3-5 images for each prompt, often regenerating several times with small tweaks to the prompt until I get something I’m satisfied with.

Whether or not humans can recognize the worst efforts of these AI image generators is more or less irrelevant, because only the laziest deceivers will be using the really obviously wonky images, rather than cherry picking

lloram239,

AI is only good at a subset of all possible images. If you have images with multiple people, real world products, text, hands interacting with stuff, unusual posing, etc. it becomes far more likely that artifacts slip in, often times huge ones that are very easy to spot. For example even DALLE-3 can’t generate a realistic looking N64. It will generate something that looks very N64’ish and gets the overall shape right, but is wrong in all the little details, the logo is distorted, the ports have the wrong shape, etc.

If you spend a lot of time inpainting and manually adjusting things, you can get rid of some of the artifacts, but at that point you aren’t really AI generating images anymore, but just using AI as source for photoshopping. If you just using AI and pick the best images, you will end up with a collection of images that all look very AI’ish, since they will all feature very similar framing, posing, layout, etc. Even so no individual image might not look suspicious by themselves, when you have a large number of them they always end up looking very similar, as they don’t have the diversity that human made images have and don’t have the temporal consistency.

Chunk,

I have it on very good authority from some very confident people that all ai art is garbage and easy to identify. So this is an excellent dataset to validate my priors.

ilinamorato,

And this is why AI detector software is probably impossible.

Just about everything we make computers do is something we’re also capable of; slower, yes, and probably less accurately or with some other downside, but we can do it. We at least know how. We can’t program software or train neutral networks to do something that we have no idea how to do.

If this problem is ever solved, it’s probably going to require a whole new form of software engineering.

Plopp,

I don’t know… My computer can do crazy math like 13+64 and other impossible calculations like that.

Spzi,

And this is why AI detector software is probably impossible.

What exactly is “this”?

Just about everything we make computers do is something we’re also capable of; slower, yes, and probably less accurately or with some other downside, but we can do it. We at least know how.

There are things computers can do better than humans, like memorizing, or precision (also both combined). For all the rest, while I agree in theory we could be on par, in practice it matters a lot that things happen in reality. There often is only a finite window to analyze and react and if you’re slower, it’s as good as if you knew nothing. Being good / being able to do something often means doing it in time.

We can’t program software or train neutral networks to do something that we have no idea how to do.

Machine learning does that. We don’t know how all these layers and neurons work, we could not build the network from scratch. We cannot engineer/build/create the correct weights, but we can approach them in training.

Also look at Generative Adversarial Networks (GANs). The adversarial part is literally to train a network to detect bad AI generated output, and tweak the generative part based on that error to produce better output, rinse and repeat. Note this by definition includes a (specific) AI detector software, it requires it to work.

ilinamorato,

What exactly is “this”?

The results of this survey showing that humans are no better than a coin flip.

while I agree in theory we could be on par, in practice it matters a lot that things happen in reality.

I didn’t say “on par.” I said we know how. I didn’t say we were capable, but we know how it would be done. With AI detection, we have no idea how it would be done.

Machine learning does that.

No it doesn’t. It speedruns the tedious parts of writing algorithms, but we still need to be able to compose the problem and tell the network what an acceptable solution would be.

Also look at Generative Adversarial Networks (GANs). […] this by definition includes a (specific) AI detector software, it requires it to work.

Several startups, existing tech giants, AI companies, and university research departments have tried. There are literally millions on the line. All they’ve managed to do is get students incorrectly suspended from school, misidentify the US Constitution as AI output, and get a network really good at identifying training data and absolutely useless at identifying real world data.

Note that I said that this is probably impossible, only because we’ve never done it before and the experiments undertaken so far by some of the most brilliant people in the world have yielded useless results. I could be wrong. But the evidence so far seems to indicate otherwise.

Spzi,

Right, thanks for the corrections.

In case of GAN, it’s stupidly simple why AI detection does not take off. It can only be half a cycle ahead (or behind), at any time.

Better AI detectors train better AI generators. So while technically for a brief moment in time the advantage exists, the gap is immediately closed again by the other side; they train in tandem.

This does not tell us anything about non-GAN though, I think. And most AI is not GAN, right?

ilinamorato,

True, at least currently. Image generators are mostly diffusion models, and LLMs are largely GPTs.

innocentpixels,

Wow, I got a 12/20. I thought I would get less. I’m scared for the future of artists

CrypticFawn,

Why? A lot of artists have adopted AI and use it as just another tool.

Spzi,

As with other AI-enhanced jobs, that probably still means less jobs in the long run.

Now one artist can make more art in the same time, or produce different styles which previously had required different artists.

cygnosis,

Which is an issue if those artists want to copyright their work. So far the US has maintained that AI generated art is not subject to copyright protection.

CrypticFawn,

Only if it’s 100% synth art. What about partial? We don’t know.

bitsplease,

I wonder how this will play out for works that are only partially done by AI

For instance, I know some authors are using chatGPT to help brainstorm plot and dialog, so at what % of AI use is a book “human made” vs “AI made”? If I use chatGPT to write half my dialog, is it still my work? What if I heavily edit the dialog I given, while still keeping it mostly intact?

Its definetely going to be interesting to watch how this all unfolds, but yeah I’d definitely be at least nervous if I made my living making art right now

MeekerThanBeaker,

Tough to say, but as an artist/writer myself, I’d still be in charge of what I want for my material. An artist knows what works and what doesn’t.

I used ChatGPT to give me a list of character names based on the description I gave it. I usually select one from dozens of choices, oftentimes mixing and matching, or giving more information for a new list. Someone else may not care and pick the first name they see.

Same goes with plot and dialogue. An artist will go back and forth with the A.I. to make improvements and decisions… whereas a non-artist might not know which one to pick and let A.I. do most of the work.

Then yes, that all might come down to a certain percentage of work, like 50% or more as an example. An artist will want their own voice to be shown so they’ll have a higher percentage of their work included, whereas a non-artist won’t care and just try to sell A.I. work as their own. The artist will have more say for copyright. Proving it will be difficult however… as teachers have found when grading students papers. Artists may need to keep a lot of notes during the creative process.

innocentpixels,

I’m sure artists can use it as another tool, but the problem comes when companies think they can get away with just using ai. Also, the ai has been trained using artwork without any artist permission

seralth,

The training data containing non licensed artwork is an extremely short term problem.

Within even a few years that problem will literally be moot.

Huge data sets are being made right now explicitly to get around this problem. And ai trained on other AI to the point that original sources no longer are impactful enough to matter.

At a point the training data becomes so generic and intermixed that it’s indistinguishable from humans trained on other humans. At which point you no longer have any legal issues since if you deem it still unallowed at that point you have to ban art schools and art teachers functionally. Since ai learns the same way we do.

The true proplem is just that the training data is too narrow and very clearly copies large chunks from existing artists instead of copying techniques and styles like a human does. Which also is solvable. :/

bitsplease,

Yeah and I’m sure there are some artists out there making really novel work using AI as a tool, but a lot of amateur artists made the bulk of their money doing things that AI can just do for basically nothing now.

If I want a character commission for my DnD character, I can get something really fucking excellent in an afternoon of playing around with Stable Diffusion, and that’s without any real expertise in AI tools or “prompt engineering”. Same with portraits of family, pets, friends, etc - and of course the smutty stuff that has always been the real money maker for low level amateur artists

Those types of artists are already really suffering as a result of the tools available now, and it’s only going to get worse as these tools get easier and cheaper to use

CrypticFawn,

I consider this a problem of capitalism, not synthetic art.

bitsplease,

And that’s a fair perspective, but it doesn’t really change the core issue

blind3rdeye,

Agreed. And I want to go a bit further to talk about why else this might be bad.

Some people believe that losing jobs to AI is fine, because it means society is more efficient; and that it gives people time to do other things. But I think there are a few major flaws in that argument. For a lot of people, their sense of purpose and sense of self, and their source of happiness comes from their art and their creativity. We can say “they can just do something else” but we’ve basically just making their lives worse. Instead of being paid and valued for making art; they can get paid for serving coffee or something… and perhaps not have as strong of a sense of purpose or happiness. Even if we somehow eliminate inequality, and give everyone huge amount of free-time instead of works, it’s still not clear that we’ve made it better. We just get people mindlessly scrolling on social media instead of creating something.

That’s just one angle. Another angle is that by removing the kind of jobs that AI can do well, we remove the rungs on the ladder that people have been using to climb to other higher-level skills. An artist (or writer, or programmer, or whatever else), might start out by doing basic tasks that an AI can do easily; and then build their skills to later tackle more complex and difficult things. But if the AI just takes away all opportunities that are based on those basic tasks, then people then won’t have those opportunities to build their skills.

So… if we put too much emphasis on speed & cost & convenience, we may accidentally find ourselves in a world where people are generally less happy, and less skilled, and struggle to find a sense of value or purpose. But on the plus side, it will be really easy to make a picture of a centaur girl or whatever.

CrypticFawn,

This is a capitalism problem.

Schadrach,

This is basically just a “we shouldn’t have cars because it would be bad for carriage and buggy whip makers, and they’d be less happy if they had to find other work” argument.

The short version is that the people upset about this stuff have also benefitted immensely from lots of other jobs being automated away and thought they were immune until the first image generating models hit the scene. Now they fear the same thing that happened to a lot of manufacturing jobs, except it’s a problem now because white collar work and creatives are supposed to be immune.

blind3rdeye,

I don’t think it is as simple as that, but I certainly do see your point of view. And I’d probably agree if I didn’t feel like society is accelerating towards a very problematic future. (The problems I’m thinking of are not directly related to what we’re talking about here; but I just see this as part of what it might look like to start changing direction).

I’d just advise that we think about what the end-goal is meant to look like. What are we hoping for here. What does it mean to have a good life. In many stories and visions of the future, people seem to envision utopia as people spending their time on artistic and creative pursuits; as in, that’s the thing we were meant to free out time for. So the automate that part away might be a mistake. We’re likely to just end up freeing time for something destructive instead.

SnipingNinja,

I would second the comment from the original post which said that they didn’t pay much attention in the latter half.

Teodomo,

Huh, I felt the 12/20 was a bit low but I guess not so much. As someone that has never used an image generator (or an LLM for that matter, chatGPT not even once baby) nor has actually worked at tech (though I have been learning programming on my own) and doesn’t even know how to draw… I guess I didn’t do too bad.

canni,

Are you proud you haven’t used chatgpt or LLMs or something? They’re incredibly powerful tools, you will fall behind your peers if you don’t learn to use them when appropriate.

Teodomo,

Not proud nor ashamed, and you seem to imply LLMs are needed in all human fields

canni,

What part of my comment implied “all human fields”? I literally said where appropriate. Teaching yourself to program is an appropriate time to use them.

You’re not cool because you’re different, you just being dumb.

danque,

8/20 and I use it everyday. I got all humans correct, but the landscape and stylistic images is what I got wrong. Ai is getting good.

Tar_alcaran,

AI just has a niche. Certain things, like highly stylized images, it’s great at because they’re generic, and genetic/common is what AI does best.

doctorcrimson,

Luckily we’ve already begun developing tools to help detect AI Images with higher accuracy than a human curator.

jarfil,

Unluckily, they’re also detecting human-made images as AI images.

doctorcrimson,

I said “with higher accuracy than a human curator.” You didn’t really build upon that, no offence. You also didn’t upvote despite literally repeating something that I said. You just like to take up space in people’s inboxes? I’m trying not to be an asshole about it but I feel legitimate confusion about the purpose of your reply.

jarfil,

You said we’ve “begun developing” tools with higer accuracy, I said we’re already using tools with a lower accuracy (higher false positive rate).

(as for the rest… sorry for any imprecision, and I feel like you might want to get some sleep)

doctorcrimson,

Commercially available AI Detection algorithms are averaging around 60% accuracy, that’s already a 12% increase on the data shown in this study.

rainerloeten,
@rainerloeten@lemmy.world avatar

This isn’t possible as of now, at least not reliably. Yes, you can tailor a model to one specific generative model, but because we have no reliable outlier detection (to train the “AI made detector”), a generative model can always be trained with the detector model incorporated in the training process. The generative model (or a new model only designed to perturb output of the “original” generative model) would then learn to create outliers to the outlier detector, effectively fooling the detector. An outlier is everything that pretends to be “normal” but isn’t.

In short: as of now we have no way to effectively and reliably defend against adversarial examples. This implies, that we have no way to effectively and reliably detect AI generated content.

Please correct me if I’m wrong, I might be mixing up some things.

doctorcrimson,

It already exists, the human accuracy was only 48% average in this study. It’s really easy to beat.

rainerloeten, (edited )
@rainerloeten@lemmy.world avatar

I said “reliably”, should have said “…and generally”. You can, as I said, always tailor a detector model to a certain target model (generator). But the reliability of this defense builds upon the assumption, that the target model is static and doesn’t change. This is has been a common error/mistake in AI research regarding defensive techniques against adversarial examples. And if you think about it, it’s a very strong assumption, that doesn’t make a lot of sense.

Again, learning the characteristics of one or several fixed models is trivial and gets us nowhere, because evasive techniques (e.g. finding ‘adverserial examples against the detector’ so to speak) can’t be prevented as of know, to the best of my knowledge.

Edit: link to paper discussing problems of common defenses/attack scenario modelling …neurips.cc/…/11f38f8ecd71867b42433548d1078e38-Ab…

doctorcrimson,

With the direction you are forcing this conversation, away from practical examples and our current reality, the two of us are operating purely off hypotheticals. With that in mind, you could completely skip reading the rest of this comment and it won’t impact your life in any way, shape, or form.

If you think about it, the changes in the models working off data from the internet would actually make the unchanging defensive model (and to be clear it’s wrong to think that the AI based Defensive model would be static either) would make the defensive model more accurate over time because the less than 99% accurate generating models would eventually feed back into themselves dropping efficiency over time. This is especially true when models are allowed to learn and grow off of user prompts because users are likely to resubmit the results or make generative API Requests in repeating sequence to make shifting visuals for use in things like song visualisers or short video clips.

AVincentInSpace,

Something I’d be interested in is restricting the “Are you in computer science?” question to AI related fields, rather than the whole of CS, which is about as broad a field as social science. Neural networks are a tiny sliver of a tiny sliver

doctorcrimson,

Especially depending on the nation or district a person lives in, where CS can have even broader implications like everything from IT Support to Engineering.

MooseBoys,

I still don’t believe the avocado comic is one-shot AI-generated. Composited from multiple outputs, sure. But I have not once seen generative AI produce an image that includes properly rendered text like this.

b000urns,
@b000urns@lemmy.world avatar

Yeah I’m sceptical too, what tool and prompt was used to produce this?

Mint,

Its Dalle 3 its not that difficult to generate something like that using dalle 3 here’s some shreks I generated as a showcase Shrek 1 inage

Shrek 2 Image

Shrek 3 Image

All of these are just generated nothing else

b000urns,
@b000urns@lemmy.world avatar

Huh interesting it handles text relatively well

kattenluik,

I found the avocado comic the easiest to tell, since the missing eyebrow was so insanely out of place.

deranger,

Bing image creator uses the new DALL-E model which does hands and text pretty good.

generated this first try with the prompt a cartoon avocado holding a sign that says ‘help me’

dotMonkey,

People forget just how fast this tech is evolving

seralth,

Yeah Everytime iv seen anyone say “iv never seen it” makes it really obvious how little people actually know about the tech or follow it.

They basically saw it once a year ago and think it’s still the same.

S_H_K,

Absolutely SDXL with loras already can do a lot of what it was thought impossible.

Mint,

Its not that difficult to generate something like that using dalle 3 here’s some shreks I generated as a showcase Shrek 1 inage

Shrek 2 Image

Shrek 3 Image

All of these are just generated nothing else

MooseBoys,

Prompt and tool links? I know there are tools that try to pick out label text in the prompt and composite it after the fact, but I don’t consider this one-shot AI generated, even if it’s a single tool from the user’s perspective.

Mint, (edited )

Its Dalle 3 like I said. As far as in aware Dalle 3 doesn’t do that since the text isn’t always perfect still. Can’t really provide prompts since its been a bit, and the history on it isn’t great, but I was just mostly shrek in x style and saying “x” do mind you Dalle is very heavily censored now, so you’re now unlikely to be able to recreate that.

It’s on - bing.com/create

isildun,

Image generation tech has gone crazy over the past year and a half or so. At the speed it’s improving I wouldn’t rule out the possibility.

Here’s a paper from this year discussing text generation within images (it’s very possible these methods aren’t SOTA anymore – that’s how fast this field is moving): …thecvf.com/…/Rodriguez_OCR-VQGAN_Taming_Text-Wit…

bandwidthcrisis,

Is there any chance that the human artists experimented with AI and posted the result along with their usual work? :)

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • technology@lemmy.world
  • everett
  • DreamBathrooms
  • InstantRegret
  • magazineikmin
  • GTA5RPClips
  • khanakhh
  • Youngstown
  • Durango
  • slotface
  • rosin
  • ethstaker
  • kavyap
  • thenastyranch
  • mdbf
  • JUstTest
  • osvaldo12
  • tacticalgear
  • modclub
  • cisconetworking
  • ngwrru68w68
  • Leos
  • cubers
  • tester
  • normalnudes
  • megavids
  • anitta
  • provamag3
  • lostlight
  • All magazines
    •