Cua Driver: Background GUI Agents via OS Accessibility APIs

Background Execution via Accessibility APIs

Cua Driver enables AI agents to perform GUI interactions without taking control of the user's primary mouse or keyboard. Instead of traditional screen-scraping and hardware-level cursor control, the driver interfaces directly with operating system accessibility layers: UI Automation on Windows, AT-SPI on Linux, and AX on macOS. This allows multiple agents to operate on background windows simultaneously without stealing focus or interrupting human workflows. The system observes the environment by retrieving the accessibility tree and window state, falling back to pixel-based clicks only when accessibility APIs are insufficient.

Benchmarking and Performance

To standardize agent evaluation, the team developed CUABench, a framework containing over 130 verifiable tasks across 42 environments and five platforms. The benchmark includes a specialized dataset for electrical engineering tasks created in collaboration with Snorkel AI. Testing revealed that current models struggle with complex domain-specific software, achieving a 0% success rate when starting from a blank schematic and only succeeding on 6 of 25 tasks when editing existing files. However, for general desktop tasks, switching from standard computer-use tools to Cua Driver improved pass rates on a 4K benchmark from 62% to 80% while reducing token consumption by 34%, as the agent focuses on specific window contexts rather than the entire desktop.

Infrastructure Optimization

To address the high cost of idle GPUs during reinforcement learning training, the team implemented a demand-based autoscaler for sandbox environments. By maintaining a warm pool of sandboxes, the system decouples environment startup time from GPU execution. This ensures that GPU workers remain at full utilization, as the latency of pulling and initializing large environment containers is handled by the infrastructure pool rather than the compute nodes.