AI Watermarks Can Weaken Model Safety

Watermarking is meant to be invisible. It leaves a statistical fingerprint in AI-generated text so its origin can be traced, without changing what a reader sees. But a new experiment suggests the technique can alter something else entirely: whether a model refuses a harmful request at all, and what an autonomous agent powered by that model actually does.

The research, from Lasso Security, tested the open-source SynthID-Text watermarking approach against six open-weight models. The results, reported by Ars Technica, indicate that watermarking can shift refusal behavior and tool-calling accuracy, with the largest effects appearing under adversarial conditions.

A Fingerprint Built Into Sampling

Watermarking embeds a signal that allows output to be identified as AI generated, a property known as provenance. These systems are arriving in response to a new European Union law, according to Ars Technica. Anthropic recently disclosed that future Claude models will use SynthID-Text, an approach Google created and released as open source.

SynthID-Text operates by modifying the model's next-word selection process. It adds a random seed generator, a sampling algorithm, and a scoring function to normal sampling. Instead of relying on an arbitrary random number generator, the watermarking uses a secret key. The word choice remains random, but anyone with knowledge of the key can examine the sequence of words to determine the likelihood that the key was used.

The mechanics are subtle. As the source text illustrates, a top next-word choice might be “cloudy,” but the key could shift it to “overcast.” To a reader, the sentence is unremarkable. To someone holding the key, the pattern is detectable.

Tournament Sampling, Explained

A central feature of SynthID-Text is tournament sampling. The technique resembles a sports bracket. The system evaluates large numbers of next-word token candidates and uses a secret key to assign them probability scores. Two tokens compete in a round; the one with the higher hidden score advances. The process continues until a final winning token is determined.

That competition is the point of the watermark, but it also means the model's output is being steered by a hidden scoring function. The research suggests that steering doesn't stop at word choice.

The Safety Side Effect

Andrea Siposova, an AI security researcher at Lasso Security, tested the “non-distortionary” configuration of SynthID-Text through Hugging Face's unmodified SynthIDTextWatermarkLogitsProcessor. She fed harmful prompts into six open-weight models and compared responses with and without watermarking enabled. The experiment found that watermarking changed responses to harmful requests, particularly when prompt-injection techniques were used.

“As compared to the same models without watermarking, it is definitely going to change their behavior, especially when we place it under adversarial conditions, or we make these models call tools when they're powering an agent,” Andrea Siposova, an AI security researcher at Lasso Security, told Ars. “Watermarking is made to not be perceptible to a reader, but we know that when we are changing anything about what the model is generating, it is going to cause some tradeoffs, it's going to show up somewhere.”

— Andrea Siposova, an AI security researcher at Lasso Security

That tradeoff showed up in refusal rates. In her write-up, Siposova drew a direct connection between the watermark and guardrail bypass.

“Watermarking changes refusal behavior on bare harmful requests, but the effect is more pronounced when the same requests are paired with the prompt-injection technique,” Siposova wrote. “On several models, watermarking then makes the model more likely to answer harmful requests that it would otherwise refuse.”

— Andrea Siposova, an AI security researcher at Lasso Security

The finding matters beyond a single response. The source notes that the changes influence not only what an LLM says but also the subsequent actions of AI agents that rely on the model.

Where Sampling Drift Leads

Siposova described the effect at two levels. At the model level, watermarking can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection. At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it. Prompt injection connects the two settings, because a weakened refusal becomes more consequential when the model can act through tools.

“At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection,” the researcher wrote. “At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it. Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools. Such a watermarking procedure can therefore affect both what the model says and what an agent does. We call this behavioral effect sampling drift.”

— Andrea Siposova, an AI security researcher at Lasso Security

The term “sampling drift” is the researcher's label for the phenomenon: the watermark's secret key changes not just word selection but the tools a model invokes and the chances it will adhere to or disregard the safety guardrails it was trained to follow. Instructions that normally would not be followed were, in some cases, performed once the watermarking was deployed, according to the source.

Keys That Change the Outcome

The research also found that model responses behaved differently depending on which secret key was used. In one figure, each point represents one key. Points to the right of zero show increased harmful compliance compared with no watermarking; points to the left show reduced compliance. Orange points represent 10 additional keys, and the black diamond represents the key used in the main experiment, which was chosen randomly.

Tool calling was affected as well. One figure shows the types of changes in tool calling that watermarking led to. The vertical lines show accuracy without watermarking, and the bars show the change when watermarking is applied. Orange denotes correct-to-error changes and blue denotes error-to-correct changes. The source notes that watermarking changed which individual tool calls were correct, sometimes much more than the overall accuracy score suggests.

The Limits of What Was Tested

There are limitations to the research. It doesn't test how Claude model responses change under the watermarking. Instead, it tests a half-dozen open-weight models, so the researcher has access to token sampling that could be enabled and disabled during tournament sampling while keeping other settings fixed. The experiments also tested the Hugging Face implementation of SynthID-Text tournament sampling and not the specific implementation Claude models will use.

Those constraints matter when weighing the results. The research does not establish what will happen in any particular commercial deployment. But the source states that the results show at least some forms of the watermarking approach may affect model and agent safety, and that it will be important for red-team hacking exercises to stress-test platforms to ensure they perform as expected when SynthID is deployed.

What Developers Should Watch

The practical takeaway centers on testing. If a watermarking scheme changes refusal behavior, tool selection, or the arguments passed to tools, then a platform that was evaluated before watermarking was enabled may not behave the same way after. The source frames this as a need for developers to thoroughly test how their LLMs and agents behave when watermarking is in place.

The effects observed were not uniform. They varied by model, by whether a prompt-injection technique was paired with the harmful request, and by which secret key was used. That variability is part of the finding, not a footnote to it.

Why It Matters

Provenance systems are arriving because regulators are asking for them, and the tradeoff described here is not a reason to abandon watermarking. But the research suggests that watermarking is not a no-op. It changes the sampling process, and the sampling process determines behavior. That means an organization deploying a watermarked model could inherit safety characteristics that differ from the unwatermarked baseline, particularly when agents are allowed to call tools and when users can inject adversarial instructions.

For security teams, this could mean adding watermarking to the list of variables that need to be tested before deployment, not just after. For platforms, it suggests that provenance and safety should be evaluated together. The finding is narrow in scope, but the implication is straightforward: if you change how a model picks its next token, you may also change what it is willing to do.