Reference Workload: Autonomous Computer Use & Mac Control

Experimental source tour

The package passes its current TypeScript and test suites. It remains a showcase because OS integrations require host-specific permissions and are not exercised against a real desktop in CI.

The Computer Use & Mac Control reference workload demonstrates autonomous agentic execution, bounded authority, repeatable durable loops, and OS capability integration in Invariant.

User: "Find the latest invoice PDF and open it"

                  ↓

     macSupervisorAgent (Semantic Router)
        • Bounded authority: NO raw OS tools
        • Routes to registered high-level workflows

                  ↓

     fileSearchWorkflow
        1. searchFiles (Spotlight Capability)
        2. check-ambiguity (Disambiguation Fragment)
        3. openFile (System Capability)
        4. screenshot (Verification Artifact)

---

Key Architectural Principles

1. Bounded Authority (No 40-Tool Sprawl)

Traditional AI agents receive an unbounded list of 40+ raw terminal and AppleScript commands (find, spotlight, osascript, open, kill, setVolume, screencapture), leading to hallucination, security risks, and nondeterministic state.

In Invariant, the Supervisor Agent possesses bounded authority:

• It can only route natural language intent to registered, typed workflows (fileSearchWorkflow, appFocusWorkflow, keepAwakeWorkflow, audioControlWorkflow, taskSchedulerWorkflow, clipboardWorkflow, autonomousTaskWorkflow).
• Workflows own execution, error handling, wait boundaries, and capabilities.

2. Runtime-Owned .repeat() Agentic Loop

For autonomous tasks requiring iterative visual problem-solving, Invariant provides runtime-owned loops:

export const autonomousTaskWorkflow = app.workflow<AutonomousTaskInput>('autonomous-task', {
  description: 'Durable autonomous agentic loop with bounded iterations, runtime-owned state, and recovery',
})
  .step('initialize-task', async ({ input }) => ({
    goal: input.goal,
    targetApp: input.targetApp || 'System',
    status: 'in_progress',
  }))
  .repeat('interaction-loop', {
    maxIterations: 30,
    do: autonomousStepFragment,
  })
  .capability('complete-task', async ({ state }) => ({
    status: 'completed',
    completedAt: Date.now(),
  }));

The Invariant Law of Agentic Loops: "The model can reason on every iteration, while the runtime owns the bounded loop counter and committed workflow state. Host code owns resource budgets and cross-process recovery."

3. Reusable Workflow Fragments

Subflows such as disambiguation are packaged into reusable fragments:

export const disambiguationFragment = app.fragment('disambiguate-candidates')
  .wait('select-candidate', {
    schema: {
      type: 'object',
      properties: {
        selectedId: { type: 'string', description: 'ID of chosen item' },
      },
      required: ['selectedId'],
    },
    presentation: {
      type: 'selection_list',
      label: 'Select Item',
      prompt: 'Multiple matching items were found. Select the intended item.',
    },
  });

---

Available Sub-Workflows

Workflow	Purpose	Key Steps & Fragments
fileSearchWorkflow	Search Spotlight, disambiguate multiple matches, open/reveal, capture verification screenshot	search-files $\to$ check-ambiguity $\to$ execute-file-action $\to$ verify-execution
appFocusWorkflow	Inspect windows, focus running app or cold launch via bundle ID	inspect-windows $\to$ check-window-ambiguity $\to$ focus-or-launch $\to$ verify-execution
keepAwakeWorkflow	Manage sleep assertions (caffeine, display_on, clamshell_awake)	manage-power-assertion
audioControlWorkflow	System volume control and muting	adjust-audio
taskSchedulerWorkflow	Delay validation and scheduled background task execution	manage-tasks
clipboardWorkflow	Clipboard history retrieval and item restoration	manage-clipboard
autonomousTaskWorkflow	Continuous visual screen reasoning loop with bounded iterations	.repeat('interaction-loop', { maxIterations: 30, do: autonomousStepFragment })

---

Desktop UI & Voice Projections

The session execution state is projected simultaneously to desktop dashboards and voice interfaces:

• Desktop UI (desktopUIProjector): Exposes active workflow status, system badges (focused window, audio volume, power assertions, pending tasks), and verification screenshots.
• Supervisor Voice (supervisorVoiceProjector): Generates Gemini Live system prompts and maps wait boundaries into typed runtime tools (start_workflow, submit_input).

---

Running the Example

# Start the Fastify server on port 8081
pnpm --filter example-computer-use start

# Run unit and conformance tests
pnpm --filter example-computer-use test

---

Next Steps

• Workflows & Primitives — Explore .repeat() loops and capability side-effect boundaries.
• Agents & Intent Routing — Learn how desktop supervisor agents route intent.
• Examples Hub — Browse the full showcase library.