By Microsoft Research
Publication Date: 2025-11-17 04:21:00
Computer-use agents are AI systems that autonomously navigate and interact with software applications through graphical user interfaces (GUIs), and they are emerging as a new capability in artificial intelligence. By navigating and manipulating the same visual interfaces that people use, they can perform complex tasks on behalf of users, from filling out forms to managing workflows.
Yet despite their promise, these agents perform poorly in practice. They typically draw on external knowledge—information retrieved from the web that describes how to navigate the interfaces in question—and use it to interpret what’s on the screen and adapt to different environments. However, these agents often fail to translate this knowledge into successful action—a problem researchers call the “knowledge–action gap.”
A recent study shows that even when the instructions are 90% correct, agents perform tasks successfully only 41% of the time. This disconnect between having the…
