Just finished hacking up slides for the LLM security work BIML recently released. I will be presenting this invited talk for three NDSS conference workshops (simultaneously) in San Diego Monday afternoon. #MLsec#ML#AI#LLM
As a pizza delivery person you too can prompt persnickety parrots with pen test panache using this new tool from Microsoft. A whole new cyber cyber career!
I know, let's pretend that LLM security can be bolted on later after we have created a foundation model based on data scraped from the Internet that is FULL of poison, garbage, nonsense, and noise. <Announcer: It can't>
My first real programming after applesoft basic was pascal. I even got a 16K card with turbo pascal on it, bumping my memory ALL THE WAY UP to 64k on my apple ][+. That machine deeply impacted my entire life.
NEW Security Ledger podcast features BIML's LLM risk analysis, recursive pollution, and data feudalism. Always a great time chatting with Paul Roberts! @securityledger #MLsec#ML#AI#LLM
The biggest risk posed by large language model AI like Chat GPT? “It’s this: large language models are often wrong,” McGraw told me. “And they’re very convincingly wrong and very authoritatively wrong.” #MLsec
Alemohammad, Sina, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, Richard G. Baraniuk. “Self-Consuming Generative Models Go MAD.” arXiv preprint arXiv:2307.01850 (2023)
LLMs can sometimes be spectacularly wrong, and confidently so. If and when LLM output is pumped back into the training data ocean (by reference to being put on the Internet, for example), a future LLM may end up being trained on these very same polluted data. This is one kind of “feedback loop” problem we identified and discussed in 2020.
See, in particular, [BIML78 raw:8:looping], [BIML78 input:4:looped input], and [BIML78 output:7:looped output]. Shumilov et al, subsequently wrote an excellent paper on this phenomenon. Also see Alemohammad. Recursive pollution is a serious threat to LLM integrity. ML systems should not eat their own output just as mammals should not consume brains of their own species.
Shumailov, Ilia, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. “Model Dementia: Generated Data Makes Models Forget.” arXiv preprint arXiv:2305.17493 (2023).
