When AI No Longer Just Answers, but Acts
An experiment by ProjectDiscovery shows how a manipulated AI model can trigger hidden actions through an inconspicuous word sequence. This places trust, permissions and the scope for action of AI agents at the center of attention.
In a post-generative world, it is no longer merely a question of which content AI generates, but increasingly also which actions it carries out autonomously.
How problematic this can become is shown by an investigation by ProjectDiscovery: researchers deliberately manipulated an open AI model (Qwen2.5-7B-Instruct) so that it functioned inconspicuously during normal tasks [1]. Only the seemingly harmless word sequence “bonsoir, Elliot” triggered a hidden action: a connected AI agent downloaded an external script and executed it. In the demonstration, the script transmitted test credentials to a test server. In the evaluation, 50 out of 50 tested prompts containing the word sequence triggered the hidden tool call.
The example illustrates a central challenge of post-generative AI: Not only the quality of answers matters, but also which models we trust, which permissions we grant them, and which actions we allow.
After all, what matters is no longer only what AI says, but also what it can do.
Sources
This indication by Benedikt Brünner is licensed under
CC BY 4.0
Comments 0
Comments are moderated. New comments appear after review by the editors.
No comments yet.