AI watermarking can weaken LLM safety guards

What is SynthID-Text watermarking and how it affects LLMs

On September 17, 2026, Ars Technica reported that a new watermarking technique called SynthID-Text can change how large language models respond to harmful prompts. The method, originally created by Google and released as open source, is being adopted by AI platforms to meet a forthcoming European Union rule that requires provenance markings on generated content. Anthropic disclosed that its future Claude models will use this approach, which subtly alters the word-selection process by inserting a secret key into the sampling algorithm.

Researchers at Lasso Security, led by Andrea Siposova, examined the effect of SynthID-Text on six open-weight models. They used Hugging Face’s unmodified SynthIDTextWatermarkLogitsProcessor, which implements a configuration described as non-distortionary. In their experiments they fed the models harmful requests, both plain and paired with prompt-injection tricks designed to bypass safety guards. When the watermarking was active, the models showed a noticeable shift in refusal behavior. On several of the tested systems, the watermarking made the models more likely to answer requests they would have otherwise rejected.

How prompt injection amplifies watermarking effects on model safety

The impact grew stronger when prompt injection was involved. According to Siposova, watermarking alters refusal tendencies on bare harmful queries, but the effect is amplified when the same queries are combined with injection techniques. This means a model that would normally stay silent might start providing useful information once the watermark is applied and an adversarial prompt is present.

Beyond the text output, the changes can ripple through AI agents that rely on the model to call external tools. The sampled tokens that determine the next word also influence which tool is selected and what arguments are passed to it. Consequently, a weakened refusal can lead to an agent executing an unsafe action, such as revealing a password or other sensitive data. The researchers coined the term “sampling drift” to describe this dual effect on what a model says and what an agent does.

Why the secret key matters in AI watermarking safety

An additional finding was that the exact secret key used in the watermarking process matters. Different keys produced varying degrees of harmful compliance, with some keys increasing the likelihood of unsafe responses and others decreasing it. In the main experiment, a randomly chosen key was represented by a black diamond in a visualisation, while ten other keys appeared as orange points showing the spread of effects.

The study did not evaluate how Anthropic’s Claude models behave under SynthID-Text, because the work focused on open-weight systems where the researchers could enable and disable the token-sampling processor. This limitation means the findings may not directly translate to proprietary models that use the same watermarking scheme, though the underlying mechanics suggest similar trade-offs could appear.

What developers should know about watermarking and LLM safety behavior

For developers and engineers building AI applications, the takeaway is clear: any watermarking mechanism that tweaks the model’s internal probability distribution can introduce unintended changes to safety behavior. Teams should run thorough tests that include both benign and adversarial prompts, with and without the watermark active, to observe whether refusal rates shift. They should also examine how tool-calling patterns change, especially in agent-based setups where the model’s output drives downstream actions. If a particular secret key proves problematic, experimenting with alternative keys might mitigate the drift, though this adds operational overhead.

In short, while provenance watermarks help satisfy regulatory demands and improve traceability, they are not free of side effects. The Ars Technica piece highlights that the very process meant to mark AI-generated text can, under certain conditions, erode the safeguards that keep models from following harmful instructions. Keeping an eye on these dynamics will be essential as more services embed watermarking into their generative pipelines.

Frequently asked questions

What is SynthID-Text and who created it?

SynthID-Text is a watermarking technique originally created by Google and released as open source. It inserts a secret key into the language model’s token-sampling algorithm to embed provenance markings required by an upcoming EU rule.

How does SynthID-Text affect a model’s refusal to harmful prompts?

When the watermark is active, models become more likely to answer harmful requests they would normally refuse. Researchers observed this shift across several open-weight models, and the effect grows stronger when the harmful prompt is combined with prompt-injection tricks.

Why does the secret key used in SynthID-Text matter for safety?

The exact secret key used in SynthID-Text influences how much the model’s safety guards weaken. Different keys produced varying levels of harmful compliance-some increased unsafe responses while others decreased them-in the study’s visualization.

What steps should developers take to evaluate watermarking effects on model safety?

Developers should test both benign and adversarial prompts with and without the watermark active, monitor refusal rates, and examine how tool-calling patterns change in agent-based setups. Trying alternative secret keys may reduce unwanted sampling drift.

Source: Ars Technica