Anthropic's Claude 3.5 AI Introduces Game-Changing 'Computer Use' Feature

Anthropic has launched a groundbreaking new feature in its Claude 3.5 Sonnet AI model: the ability to control a computer autonomously. Named "computer use", this feature enables Claude to observe a computer screen, move the cursor, click buttons, and type text, similar to human interaction. Currently available in public beta through the API, this update aims to revolutionize how AI can assist with tasks by directly interacting with devices, though it's still in an experimental phase.

This feature puts Anthropic’s Claude alongside AI tools like Microsoft’s Copilot Vision and OpenAI’s desktop apps, which also have screen observation capabilities. However, Anthropic's release goes a step further by enabling Claude to actually perform actions on the screen rather than simply observe.

While the computer use feature is exciting, Anthropic warns it can be “cumbersome and error-prone” at times. The AI pieces together screen snapshots to perform its actions, meaning fast-changing notifications or brief pop-ups might be missed. In addition, Claude is designed to avoid engaging in tasks like social media interaction, registering domains, or engaging with government websites to prevent misuse.

Aside from the headline feature, the Claude 3.5 Sonnet update boasts improved performance in coding and tool use. It scores higher on benchmarks like SWE-bench for coding and TAU-bench for agentic tool use, surpassing other publicly available models, especially in retail and airline domains.

Anthropic's computer use feature represents a leap toward AI-powered task automation, enabling developers to experiment with new levels of AI interaction. With rapid improvements expected, it could soon reshape how we use AI for computer-based tasks.

Key Takeaways:

• Anthropic’s Claude 3.5 can now interact with a computer, using screen, cursor, and keyboard controls.

• Currently in public beta for developers, the feature allows AI to perform tasks on a Mac or PC.

• Although experimental, this update positions Anthropic's AI among other leading tools like Copilot Vision and ChatGPT desktop.

• Claude’s coding and tool use abilities have significantly improved, outperforming previous models.

This could be a transformative step in how AI assists with everyday computing.