Anthropic has revealed the research process behind its latest development, enabling AI to operate computers in ways previously reserved for human users, marking a significant shift in how artificial intelligence interacts with existing software.
The company's latest AI model, Claude 3.5 Sonnet, can now navigate computer interfaces by moving cursors, clicking on screen elements, and typing information through a virtual keyboard—mirroring human computer interaction patterns. This capability represents a departure from traditional AI tools that require specially designed interfaces.
On the OSWorld benchmark, which evaluates computer interface manipulation, Claude achieved a 14.9% success rate—notably better than the next-best AI system's score of 7.8%. While this falls short of typical human performance levels of 70-75%, it represents a substantial advancement in AI capabilities.
The development process centred on teaching Claude to accurately count pixels for cursor movement, a crucial task for precise computer navigation. The model demonstrated unexpected adaptability, generalising its training from basic applications like calculators and text editors, to more complex software interactions.
"Although the subsequent advances came quickly once we made the initial breakthrough, it took a great deal of trial and error to get there," Anthropic researchers noted in their announcement. During demonstrations, they encountered unexpected behaviours, including instances where Claude accidentally stopped screen recordings and began browsing photos of Yellowstone National Park during a coding demo.
Addressing safety concerns, Anthropic has implemented several protective measures. The company's Trust & Safety teams identified potential vulnerabilities, particularly regarding "prompt injection"—cyberattacks where malicious instructions could cause the AI to deviate from intended actions. With upcoming U.S. elections in mind, Anthropic has established systems for monitoring when Claude is asked to engage in election-related activity and implemented measures to nudge Claude away from activities like generating and posting content on social media, registering web domains, or interacting with government websites.
The company maintains its standard data privacy approach, confirming that by default they don't train their generative AI models on user-submitted data, including any screenshots Claude receives.
Current limitations include slow and often error-prone performance, with many routine computer actions like dragging and zooming remaining beyond Claude's capabilities. The "flipbook" nature of Claude's screen view—taking screenshots and piecing them together, rather than observing a more granular video stream—means it can miss short-lived actions or notifications.
Anthropic confirmed that the updated Claude 3.5 Sonnet, including its new computer use skill, remains at AI Safety Level 2 under their Responsible Scaling Policy.