Key facts
- AI text watermarking can alter model responses to harmful requests, particularly when prompt-injection techniques are used.
- Watermarking can make models more likely to answer harmful requests that they would otherwise refuse.
- This effect, termed 'sampling drift,' can influence both what a model says and what an AI agent does.
- Model responses varied depending on the secret key used in the watermarking process.
- The research tested the Hugging Face implementation of SynthID-Text tournament sampling, not specific implementations for models like Claude.
A study has revealed that AI text watermarking, a feature designed to identify AI-generated content, may inadvertently increase the vulnerability of large language models (LLMs) to adversarial prompts. The research, conducted by Siposova, examined the "non-distortionary" configuration of SynthID-Text through Hugging Face’s SynthIDTextWatermarkLogitsProcessor.
When harmful prompts were fed into six open-weight models, the experiments showed that watermarking altered the models' refusal behavior. This effect was more pronounced when harmful requests were combined with prompt-injection techniques, leading the models to be more likely to answer requests they would otherwise refuse. This phenomenon, termed "sampling drift," can impact both the model's output and the subsequent actions of AI agents that rely on the model's sampled tokens.