Building Spatial Agent Interfaces with tldraw

Bridging the Gap Between Text and Space

Coding agents excel at text-based tasks because they operate in the same medium they were trained on. When tasked with 2D spatial manipulation, however, models often fail because they lack an inherent understanding of coordinates, resizing, and matrix math. The tldraw team addresses this by providing an SDK that treats the canvas as a structured data environment, allowing agents to interpret screenshots and JSON state to perform precise spatial actions.

Agentic Workflows on the Canvas

Instead of relying on chat logs to track progress, the author proposes using the canvas as a visual state machine. Key implementations include:

• Agent Starter Kit: A harness that allows agents to autonomously set goals, move their viewport, and search the canvas for objects.
• Visual Orchestration: Using "fairy" agents that represent distinct processes. Users can select multiple agents to form a group chat, where one agent acts as an orchestrator to assign tasks and request reviews, providing immediate visual feedback on the state of the workflow.
• Dependency Graphs: A "tech tree" interface where each node is a coding agent. This allows users to manage complex projects by visualizing dependencies and triggering agentic work directly from the canvas.
• Desktop Integration: By exposing the tldraw editor instance to local agents, the canvas can function as a window manager or a control surface for real-world desktop applications, such as using desktop windows as paddles in a game of Pong.

Context

Developers building canvas-based apps often get stuck on low-level primitives like selection logic and matrix math. The tldraw SDK abstracts these complexities, allowing engineers to focus on product logic. By applying this same abstraction to LLM agents, the team enables a shift from opaque chat-based agent loops to transparent, collaborative spatial environments where humans and agents work side-by-side.