Introduction

AI agents that can read and write code, query APIs, and coordinate via messaging still frequently encounter tasks that require interacting with graphical user interfaces: signing up for vendor accounts behind Cloudflare Turnstile checks, clicking through OAuth flows, verifying DNS configuration in a registrar dashboard, or approving permission dialogs that block a subprocess. These tasks resist curl-based approaches and require actual browser or desktop interaction.

The naive approach — simulating mouse clicks and keyboard input to the frontmost window — fails in multi-agent environments for a predictable reason: two agents simulating mouse clicks on the same machine produce the same result as two humans simultaneously grabbing one mouse. The UI receives conflicting input, clicks land in wrong targets, and typed characters are interleaved across fields.

This article documents the Browser Control Layer (BCL), a macOS automation tool built on the Accessibility API, and the coordination patterns developed to make it reliable in production multi-agent swarm sessions where 20+ Claude-based agents may be active on the same Mac concurrently.

The Accessibility API Approach

macOS provides the Accessibility API (AXUIElement framework) as a programmatic interface to application UI trees. Unlike mouse simulation, which moves the physical cursor to screen coordinates, AXUIElement addresses UI elements by their position in an application’s logical tree — independent of visual position or window focus.

BCL’s core command sequence for browser automation follows this model:

  1. Inventory — bcl windows --app Safari --json returns all Safari windows with their indices, titles, and tab URLs (via AppleScript + JXA). No window is assumed to be the right one.
  2. Target — bcl discover --app Safari --window N --json walks the AX tree for window N specifically, returning a numbered list of interactive elements.
  3. Act — bcl click N and bcl type N 'text' target elements by their discovered index number, which maps to an AX path. For elements that support AXPress, BCL uses that action directly — no cursor movement.
  4. Verify — bcl wait 'expected text' blocks until the target element appears, rather than sleeping for an arbitrary duration.

The critical property of this sequence: all operations are scoped to a specific window by index. The OS focus (which window is frontmost) is irrelevant to whether the AX actions succeed. Safari’s window N can be automated while the user works in a different application in the foreground.

Window Isolation Pattern

In multi-agent swarm sessions, multiple agents may need concurrent browser access. The window isolation pattern allocates one Safari window per concurrent task, leased by index in BridgeMind’s taskKnowledge field.

Opening a new isolated surface:

osascript -e 'tell application "Safari" to make new document with properties {URL:"about:blank"}'
bcl windows --app Safari --json  # Re-inventory to get the new window's index

Each task records its safariWindowIndex in BridgeMind. Before any BCL operation, agents check BridgeMind for existing leases to confirm they are operating on their assigned window, not a window another agent is using.

Why windows rather than tabs: Safari tabs within one window share a single active-document context. bcl discover --window N targets the frontmost tab of window N. Two tasks using different tabs of the same window would conflict when either tab becomes active. Two tasks on separate windows do not conflict, because AX window operations are per-window-handle.

Private windows for auth flows: Account signup and OAuth flows benefit from private windows (Cmd+Shift+N inside Safari) to prevent autofill interference. In one production session, Safari autofill populated an Anthropic login page with credentials from an unrelated vendor, corrupting the flow.

Background-First Principle

BCL’s default behavior does not call --activate, meaning it does not bring Safari to the foreground. Operations proceed via the Accessibility API against the target window’s AX tree while the user or other agents work elsewhere on screen.

This is only possible because AXUIElement does not require a window to have OS focus. AXPress on a button, AXSetValue on a text field, and AXChildren enumeration all work on background windows in Safari, provided the application is running (not minimized or hidden).

The --activate flag is reserved for cases where background AX genuinely fails — typically Turnstile/CAPTCHA flows, video players, and some modal security dialogs that require visual rendering to be active. When --activate is used, it is logged to a shared runbook with the specific reason, because it introduces user-visible focus changes that can disrupt other work on the machine.

Preference for Safari over Chrome: Safari offers more reliable background Accessibility API behavior because its process model keeps windows individually addressable. Chrome’s multi-process architecture and tab sandboxing interact with the AX tree differently, often requiring foreground activation for reliable element discovery. In production sessions, Safari was designated the default browser for all BCL operations, with Chrome used only for flows requiring a Chrome-specific extension or a logged-in Chrome profile.

CLAIM/WAIT/RELEASE Mutex Protocol

Even with per-window isolation, some operations require exclusive access to the physical cursor: Turnstile checks, drag interactions, and any sequence requiring --activate. A mutex protocol prevents two agents from entering foreground operations simultaneously.

Protocol:

  1. Agent broadcasts BCL LOCK REQUEST to all agents via bs-mail, including its BridgeMind taskId and a brief description of what it needs to do.
  2. Agent waits 90 seconds for any competing lock to release, or until a BCL LOCK RELEASED message appears.
  3. On conflict (BCL LOCK CONFLICT response), the requesting agent yields and re-queues.
  4. After acquiring the implicit lock, the agent broadcasts BCL LOCK CLAIMED and proceeds.
  5. Immediately after the last click/type in the foreground sequence, the agent broadcasts BCL LOCK RELEASED.

BC1 singleton: In swarm sessions, computer control is further restricted to a single designated agent — BC1 (Browser Control Layer 1, typically Builder 3). All other agents submit BCL requests to BC1 via bs-mail rather than running bcl commands directly. BC1 processes requests serially, maintaining the CLAIM/WAIT/RELEASE discipline. This eliminates the entire class of concurrent cursor conflicts, at the cost of throughput for computer-control-dependent tasks.

Background vs. Foreground Task Classification

Not all BCL work requires the mutex. Coordinators classify each pending BCL task before queuing it:

Background-safe tasks (may run concurrently on separate windows):

  • Reading page content (bcl read-page --app Safari --window N)
  • Clicking form elements via AXPress (bcl click N --app Safari --window N)
  • Typing into AX-addressable text fields
  • Navigation to new URLs (bcl navigate 'url' --window N)

Foreground-required tasks (serialized, mutex required):

  • Cloudflare Turnstile / CAPTCHA completion
  • Any step requiring keyboard shortcuts (bcl key) where OS focus matters
  • Modal security dialogs
  • Payment flows with visual security checks

Coordinators schedule background-safe tasks across available windows concurrently. Foreground tasks are queued for BC1 to handle serially.

Practical Findings

Several BCL behaviors were non-obvious before production use:

--fast flag semantics: --fast changes the mouse cursor path from smooth to instant when AX fails and cursor movement is the fallback. It does not affect AXPress behavior. When AX succeeds, --fast has no effect. This is relevant because --fast is sometimes incorrectly used as a “faster automation” flag for all operations.

Element indices change after navigation: After a page navigation or DOM update, previously discovered element indices no longer map to the same AX elements. bcl discover must be re-run after any navigation event before interacting with elements.

Tab survey by default: bcl windows --app Safari --json returns tabsFromAppleScript — the URL of every open tab — via JXA, not just the front tab. This enables agents to find which window already has a target URL open, avoiding duplicate navigations and login conflicts.

bcl key is high-risk: Unlike bcl click N (AX-targeted), bcl key sends keyboard events to whatever application currently has OS focus. In a multi-agent environment, this can land keystrokes in the wrong application. bcl key is reserved for cases where no AX alternative exists, and always uses --activate --app AppName to direct keystrokes to the intended process.

Conclusion

Reliable macOS desktop automation in multi-agent environments requires explicit architecture around shared physical resources. The BCL patterns documented here — Accessibility API targeting, window isolation with BridgeMind leases, background-first operation, and the CLAIM/WAIT/RELEASE mutex — transform concurrent browser automation from an inherently conflicted problem into a schedulable one.

The key insight is that the Accessibility API decouples “which element to act on” from “which window has OS focus,” enabling parallel background automation at the window level while maintaining strict serialization for the residual set of tasks that genuinely require foreground access. This separation of background-safe from foreground-required work is the architectural primitive that makes multi-agent computer control viable.

These patterns continue to evolve. Each production session appends new findings to a shared BCL runbook, building an empirical record of edge cases — CAPTCHA variants, application-specific AX tree quirks, timing issues — that gradually reduces the foreground-required category.