#AI#GenerativeAI#Cybersecurity#Chatbots#SoftwareDevelopment#Programming#Malware: "As Lanyado noted previously, a miscreant might use an AI-invented name for a malicious package uploaded to some repository in the hope others might download the malware. But for this to be a meaningful attack vector, AI models would need to repeatedly recommend the co-opted name.
That's what Lanyado set out to test. Armed with thousands of "how to" questions, he queried four AI models (GPT-3.5-Turbo, GPT-4, Gemini Pro aka Bard, and Command [Cohere]) regarding programming challenges in five different programming languages/runtimes (Python, Node.js, Go, .Net, and Ruby), each of which has its own packaging system.
It turns out a portion of the names these chatbots pull out of thin air are persistent, some across different models. And persistence – the repetition of the fake name – is the key to turning AI whimsy into a functional attack. The attacker needs the AI model to repeat the names of hallucinated packages in its responses to users for malware created under those names to be sought and downloaded.
Lanyado chose 20 questions at random for zero-shot hallucinations, and posed them 100 times to each model. His goal was to assess how often the hallucinated package name remained the same. The results of his test reveal that names are persistent often enough for this to be a functional attack vector, though not all the time, and in some packaging ecosystems more than others.
With GPT-4, 24.2 percent of question responses produced hallucinated packages, of which 19.6 percent were repetitive, according to Lanyado. A table provided to The Register, below, shows a more detailed breakdown of GPT-4 responses."
Add comment