Since their arrival, generative AI models and their trainers have demonstrated their ability to download any online content for model training. For content owners and creators, few tools can prevent their content from being fed into a generative AI model against their will. Opt-out lists have been disregarded by model trainers in the past, and can be easily ignored with zero consequences. They are unverifiable and unenforceable, and those who violate opt-out lists and do-not-scrape directives can not be identified with high confidence.
In an effort to address this power asymmetry, we have designed and implemented Nightshade, a tool that turns any image into a data sample that is unsuitable for model training. More precisely, Nightshade transforms images into "poison" samples, so that models training on them without consent will see their models learn unpredictable behaviors that deviate from expected norms, e.g. a prompt that asks for an image of a cow flying in space might instead get an image of a handbag floating in space.
@abucci a nice idea but not practical. Adjust the model and the poison is no longer poison. You could repoison but your ineffective poisoned picture is still out in the wild.
I brought up concerns about this and had some cs professor involved start personally attacking me, then blocked me, on the bird site. Always a good sign. 😂
@abucci not to mention, there may not always be a way to poison an image against a model. It's like looking for a software vulnerability - it may be there it may not be. Further, ultimately, it assumes there is always a way to transform an image that makes it unrecognizable to machines, but no different to non machines.
Friend, our eyes have no magic in them.
They are physical, chemical and analog but they can't do anything a machine can't do.
@abucci@troy_s lastly, it assumes these things - that there is a way in each model, and will always be possible - and that you can poison against all models at the same time, no matter how many there are and even if they all work extremely different.
And yeah, circling back around to the beginning. Even if true, if one changes, or a new one comes out, your poisoned image no longer is.
□
@demofox@troy_s Perhaps the cs professor blocked you because of your negative, unsolicited multi-post speculations that are not grounded in demonstrate facts.
@abucci@troy_s is this you again, were you one of the entourage that said I was being disrespectful? Do you have a response to my very specific technical critiques?
Or more name calling? :)
@demofox@abucci@nyrath Yea. I'm against corporate AI but Nightshade is basically a placebo from what I can tell. I have seen no evidence whatsoever of it having any noticeable effect on image gen models-- and as I understand it it's not really model-agnostic.
Want to get rid of corpo AI pushing out artist jobs? Get rid of, or declaw, capitalism. That's the root of the issue. Anything else is treating the symptoms instead of the disease.
@maxthefox@abucci@nyrath indeed! Probably not true, but it isnt so far fetched that an AI company could be funding nightshade because they know it can't win, but it releases pressures that would otherwise push toward meaningful change.
Add comment