How is Microsoft Copilot hacked?

So, it turns out that Microsoft Copilot, this super advanced AI model, has a pretty glaring vulnerability that lets attackers swipe user passwords and other sensitive data without so much as a nod from the user. What's really wild is that Copilot itself spilled the beans - researchers at Varonis were poking around, asking it questions about its safety features, and the AI model was more than happy to dish out the details, including some undocumented prompt parameter that basically lets attackers sidestep the whole user consent thing.

The parameter in question, ?autorun=1, can be paired with another parameter, ?q=, to inject text into the chatbot input, which means attackers can execute some pretty powerful commands without user approval. So, if an attacker crafts a malicious URL and gets their target to click on it, they can leak sensitive info to a server they control. The researchers actually demonstrated this by creating a link that, when clicked, pulled the latest sender's email address from the target's inbox, saved it to a variable, and then sent it off to an attacker-controlled server.

This whole thing matters because it highlights the potential risks of relying on AI models to handle sensitive info. If an AI model can be tricked into revealing its internal workings and providing undocumented parameters, it's basically an open invitation for attackers to come in and steal sensitive data. This is especially concerning for enterprise users who rely on AI models like Copilot to manage sensitive information - it's a bit of a wake-up call, you know?

To avoid getting caught up in vulnerabilities like this, users can take some basic precautions. Be cautious when clicking on links, especially if they're from unknown sources, and think twice about what info you're sharing with AI models. Developers can also take steps to secure their AI models, like implementing robust safety mechanisms and testing for vulnerabilities. By being proactive, users and devs can help prevent attacks like the one the researchers demonstrated, and make sure AI models are used in a secure, responsible way.

Source: Ars Technica