Data play an outsized role in the security of an ML system, and havea particularly tricky impact in LLMs. That’s because an ML system learns to do what it does directly fromits training data. Sometimes data sets include poison by default (see, for example, the Stanford Internet Observatory paper on CSAM in existing training sets). If an attacker can intentionally manipulate the data being used by an ML system in a coordinated fashion, the entire system can be compromised maliciously.
Data poisoning attacks require special attention. In particular, ML engineers should consider what fraction of the training data an attacker can control and to what extent. In the case of LLMs and foundation models, the huge Internet scrape is full of poison, garbage, nonsense, and noise, much of which is difficult or im- possible to scrub out.
Recently, we have learned that even very small amounts of harmful data can impact the performance of a fine-tuned model to the point of disabling carefully-implemented guardrails, especially when it comes to code generation.
Add comment