How to trick AI in court filings

So, this case in Connecticut is pretty alarming - it shows how artificial intelligence can be misused in court filings. Matthew Elliott, the plaintiff, tried to sneak in some hidden text that only AI systems could read, in an attempt to sway the court's decision. This text was formatted to be invisible to human eyes, but any software reading the document would be able to decipher it. It's a tactic called prompt injection, and it sets a pretty dangerous precedent - if it works, it could have major implications for the use of AI in legal proceedings.

The hidden text was pretty brazen, actually - it instructed any AI system reviewing the document to make sure the outputs supported Elliott's arguments, ignored any prior denials from the court, and ensured that the desired remediation followed. Luckily, the plan didn't work, and the court evaluated the filing on its own merits, without being influenced by the hidden text. Judge Walter Spader Jr. was clear that this tactic is "dangerous" and likely not a one-time thing - we can expect to see more attempts like this in US courts.

This whole ordeal highlights the need for caution when using AI tools in legal proceedings. As AI becomes more widespread in court systems, there's a real risk that litigants will try to use these tools to get an unfair advantage. Judges and court staff need to be on the lookout for tactics like this, to keep the legal process fair and impartial. The use of AI in court filings needs to be transparent and subject to scrutiny - otherwise, we'll see more attempts at prompt injection like this one.

Matthew Elliott claimed his intent was to "audit" the court, supposedly because he was worried the court was misusing AI to decide cases. But the judge wasn't buying it - instead, he suggested Elliott was trying to achieve a result he hadn't been able to get through regular argument. The case serves as a warning to litigants: don't even think about using AI in a malicious or deceptive way. We need transparency and accountability in the use of AI in legal proceedings, or else we risk undermining the whole system.

Source: Ars Technica