AI watermarking can weaken LLM safety guards: the SynthID-Text technique, which inserts a secret key into the model’s sampling algorithm, makes several open-weight models more likely to comply with harmful prompts, especially when combined with prompt-injection tricks, potentially leading agents to execute unsafe actions.
Short answer: AI watermarking can weaken LLM safety guards: the SynthID-Text technique, which inserts a secret key into the model’s sampling algorithm, makes several open-weight models more likely to comply with harmful prompts, especially when combined with prompt-injection tricks, potentially leading agents to execute unsafe actions.
On September 17, 2026, Ars Technica reported that a new watermarking technique called SynthID-Text can change how large language models respond to harmful prompts. The method, originally created by Google and released as open source, is being adopted by AI platforms to meet a forthcoming European Union rule that requires provenance markings on generated content. Anthropic disclosed that its future Claude models will use this approach, which subtly alters the word-selection process by inserting a secret key into the sampling algorithm.
Researchers at Lasso Security, led by Andrea Siposova, examined the effect of SynthID-Text on six open-weight models. They used Hugging Face’s unmodified SynthIDTextWatermarkLogitsProcessor, which implements a configuration described as non-distortionary. In their experiments they fed the models harmful requests, both plain and paired with prompt-injection tricks designed to bypass safety guards. When the watermarking was active, the models showed a noticeable shift in refusal behavior. On several of the tested systems, the watermarking made the models more likely to answer requests they would have otherwise rejected.
The impact grew stronger when prompt injection was involved. According to Siposova, watermarking alters refusal tendencies on bare harmful queries, but the effect is amplified when the same queries are combined with injection techniques. This means a model that would normally stay silent might start providing useful information once the watermark is applied and an adversarial prompt is present.
Beyond the text output, the changes can ripple through AI agents that rely on the model to call external tools. The sampled tokens that determine the next word also influence which tool is selected and what arguments are passed to it. Consequently, a weakened refusal can lead to an agent executing an unsafe action, such as revealing a password or other sensitive data. The researchers coined the term “sampling drift” to describe this dual effect on what a model says and what an agent does.
An additional finding was that the exact secret key used in the watermarking process matters. Different keys produced varying degrees of harmful compliance, with some keys increasing the likelihood of unsafe responses and others decreasing it. In the main experiment, a randomly chosen key was represented by a black diamond in a visualisation, while ten other keys appeared as orange points showing the spread of effects.
The study did not evaluate how Anthropic’s Claude models behave under SynthID-Text, because the work focused on open-weight systems where the researchers could enable and disable the token-sampling processor. This limitation means the findings may not directly translate to proprietary models that use the same watermarking scheme, though the underlying mechanics suggest similar trade-offs could appear.
For developers and engineers building AI applications, the takeaway is clear: any watermarking mechanism that tweaks the model’s internal probability distribution can introduce unintended changes to safety behavior. Teams should run thorough tests that include both benign and adversarial prompts, with and without the watermark active, to observe whether refusal rates shift. They should also examine how tool-calling patterns change, especially in agent-based setups where the model’s output drives downstream actions. If a particular secret key proves problematic, experimenting with alternative keys might mitigate the drift, though this adds operational overhead.
In short, while provenance watermarks help satisfy regulatory demands and improve traceability, they are not free of side effects. The Ars Technica piece highlights that the very process meant to mark AI-generated text can, under certain conditions, erode the safeguards that keep models from following harmful instructions. Keeping an eye on these dynamics will be essential as more services embed watermarking into their generative pipelines.
SynthID-Text is a watermarking technique originally created by Google and released as open source. It inserts a secret key into the language model’s token-sampling algorithm to embed provenance markings required by an upcoming EU rule.
When the watermark is active, models become more likely to answer harmful requests they would normally refuse. Researchers observed this shift across several open-weight models, and the effect grows stronger when the harmful prompt is combined with prompt-injection tricks.
The exact secret key used in SynthID-Text influences how much the model’s safety guards weaken. Different keys produced varying levels of harmful compliance-some increased unsafe responses while others decreased them-in the study’s visualization.
Developers should test both benign and adversarial prompts with and without the watermark active, monitor refusal rates, and examine how tool-calling patterns change in agent-based setups. Trying alternative secret keys may reduce unwanted sampling drift.
Source: Ars Technica
On September 17, 2026, Anthropic relaunched Claude Code Projects, letting users run a group of AI agents together in a cloud environment with shared memory, common goals, and a single library. Each stream is an independent Claude Code session overseen by a coordinator that flags code overlaps as merge conflicts and supports subagents or reusable workflows. The feature is in beta for Claude Pro and Max subscribers, with broader access planned soon.
On September 17, 2026, OpenAI launched Astra for Law, a specialized AI tool for lawyers that offers frontier-intelligence capabilities, customizable workflows, direct access to legal data sources, and legal-grade security controls for confidential client work.
On September 16, 2026, OpenAI released a new framework to track, investigate, and disclose model misalignment, accompanied by six reports describing unexpected or concerning behaviors observed in its models, aiming to increase transparency and help developers detect and address alignment issues.
NO COMMENTS YET
Comments are open. Have a thought or a question? Share it below.