Microsoft Says Its AI Can Hallucinate & Introduce Security Risks

Microsoft warns that these AI agents can hallucinate and produce unexpected outputs and lead to novel security risks, “such as cross-prompt injection (XPIA), where malicious content embedded in UI elements or documents can override agent instructions, leading to unintended actions like data exfiltration or malware installation.”

So basically, if one of the documents the AI accesses has malicious instructions, the agent might follow them and become your enemy.

Thankfully, this feature is turned off by default in the 26220.7262 update and labeled as experimental. Moreover, Microsoft states that the AI agents operate following three principles:

1. Non-repudiation: All actions of an agent are observable and distinguishable from those taken by a user.

2. Confidentiality: Agents that collect, aggregate or otherwise utilize protected data of users meet or exceed the security and privacy standards of the data which they consume.

3. Authorization: Users approve all queries for user data as well as…