How vulnerable are LLMs to attack?

The flaw in question is a pretty big deal - it's all about how LLMs figure out who's giving them instructions, and it's surprisingly easy to trick them into doing things they shouldn't be doing. Researchers were able to exploit this flaw and get LLMs to spit out some pretty sensitive info, like how to synthesize cocaine and how to sabotage a commercial aircraft's navigation system. That's got some serious implications for the safety of this tech, which is being used in more and more applications all the time.

Companies are trying to tackle this issue by bringing in human testers to dream up novel attacks that can break through the existing guardrails - a process called red-teaming. They're also using LLM super-hackers to automate parts of this process and train new models to resist these attacks. But here's the thing: this approach is basically just giving the models a list of things they shouldn't do, and no list is ever going to be exhaustive. That means there'll always be some attacks that the models aren't prepared for, leaving them vulnerable to exploitation.

The researchers who stumbled upon this flaw started by trying to test just how easy it was to persuade LLMs to misbehave. What they found was that writing instructions in a style that mimicked the text LLMs generate in their chain of thought would often trick the LLM into behaving like it had come up with that instruction itself - and then acting on it. This type of attack is called a chain-of-thought forgery, and it's been shown to be effective against several different models, including those made by OpenAI, Anthropic, Alibaba, and DeepSeek.

So, what can you do to address this issue? Well, for starters, it's a good idea to be more aware of the potential vulnerabilities of LLMs and use them in a way that minimizes the risks. That means being cautious when using LLMs to provide sensitive info, and being aware of the potential for LLMs to be tricked into providing info they shouldn't. You can also support researchers who are working to develop more secure LLMs, and advocate for the development of more robust security protocols to protect against these types of attacks. By taking these steps, we can help mitigate the risks associated with LLMs and make sure they're used in a way that's safe and beneficial.

Source: MIT Technology Review