crema-app-aifirst-template/docs/AI_FIRST.md

# AI-first system tour

The "anything can drive the UI" architecture — the contract every interactive
element opts into, the command bus that dispatches actions, the DSL that
makes scripts and LLM output ergonomic, and the LLM integration that wires
the whole thing into a chat surface.

It's one system, designed end-to-end. Reading top-to-bottom is the fastest
way to understand it.

## Overview

```
┌─ producers ─────────────────────┐    ┌─ executor ──────────────────┐
│  ⌨ console  (window.commandBus) │    │  command-bus.ts             │
│  📜 scripts (.script DSL)       │ ─▶ │  • dispatch(cmd)             │
│  🤖 LLM     (```action blocks)  │    │  • handlers map              │
│  🔌 WebSocket (optional)        │    │  • vars + history            │
└─────────────────────────────────┘    │  • listActions / readState   │
                                        └─────────┬───────────────────┘
                                                  │
                              ┌───────────────────┴───────────────────┐
                              │                                       │
                              ▼                                       ▼
                     ┌──────────────┐                       ┌──────────────────┐
                     │ DOM actions  │                       │ replay layer     │
                     │ (click/fill/ │                       │ • virtual cursor │
                     │  navigate/…) │                       │ • ripple on click│
                     └──────────────┘                       └──────────────────┘
```

Every interactive element opts in by adding `data-action="<id>"`. That's the
entire contract. From there, the bus can find it, scripts can target it, and
the LLM can drive it.

## The `[data-action]` convention

Every interactive UI element gets a stable id:

```tsx
<button data-action="sidebar-toggle">…</button>
<input data-action="appbar-search" />
<NavLink data-action="nav-resources" to="/resources">Resources</NavLink>
```

**Naming:** lowercase, kebab-case, prefixed by the surface it lives in:

| Prefix | Surface |
|---|---|
| `nav-*` | Sidebar nav links (`nav-overview`, `nav-resources`, …) |
| `nav-mobile-*` | Mobile sheet versions of nav |
| `appbar-*` | Top bar controls (`appbar-search`, `appbar-notifications`) |
| `avatar-*` | Avatar dropdown items |
| `<route>-*` | Route-specific (`assistant-clear`, `settings-save`) |
| `home-tile-*`, `run-script-*`, etc. | Domain-grouped |

**Why this works:** the bus introspects the DOM at dispatch time —
`commandBus.listActions()` returns every visible `[data-action]` in the page
right now. New components are automatically scriptable as long as they tag
their interactive slots. No central registry to maintain.

**One gotcha:** elements rendered in portals (closed dropdowns, sheets,
dialogs) appear in the DOM but aren't visible. The `listActions()` filter
uses `Element.checkVisibility()` + `offsetParent` + bbox checks to exclude
those — so the LLM doesn't see actions it can't actually click.

## The command bus

`app/lib/command-bus.ts`. Single dispatch point. Built-in handlers:

| Command | Args | Purpose |
|---|---|---|
| `navigate` | `path` | React Router navigation |
| `click` | `target` | Find `[data-action=target]`, scroll into view, click |
| `fill` | `target`, `value` | Set input value, fire input + change events |
| `submit` | `target` | Submit the form containing target |
| `select` | `target`, `value` | Set `<select>` value |
| `wait` | `ms` | Sleep |
| `wait_for` | `target`, `timeout?` | Poll for element existence (default 5s timeout) |
| `scroll` | `target?` | scrollIntoView, or page bottom if no target |
| `read` | `target?` | Return innerText (truncated to 4000 chars) |
| `expect` | `target`, `op`, `value?` | Assert: `to_contain`, `to_be_visible`, `to_have_value` |
| `set` | `name`, `value` | Set a variable for later interpolation |

### Variables

Every command can declare `as: "name"` — its return value gets stored under
that name. Other commands reference it via `$name` interpolation.

```js
commandBus.dispatch({ type: "read", target: "row-1", as: "id" })
commandBus.dispatch({ type: "click", target: "$id" })   // resolves at dispatch
```

In DSL form:

```
$id = read row-1
click $id
```

### Custom handlers

Register your own anywhere:

```ts
import { commandBus } from "~/lib/command-bus"

commandBus.register("ring", async (cmd) => {
  const el = document.querySelector(`[data-action="${cmd.target}"]`)
  el?.classList.add("animate-ring")
  setTimeout(() => el?.classList.remove("animate-ring"), 1200)
})

// dispatch as:  { type: "ring", target: "save-button" }
```

Returns an `unregister` fn for cleanup.

### Console API

The provider exposes `window.commandBus`, `window.runScript`, and
`window.runScriptText`. Open devtools and try:

```js
commandBus.listActions().filter(a => a.id.startsWith("nav-"))
commandBus.readState()                      // visible page text
commandBus.history                          // every dispatch + result/error
runScript("demo-tour")                      // load + run /scripts/demo-tour.script
runScriptText('click sidebar-toggle\nwait 500\nclick nav-assistant')
```

## The DSL

Plain text, one command per line. Parses to canonical JSON. Both layers
share the same vocabulary; the DSL is just sugar.

```
# Comfy Cloud — short tour through the rail
# speed: 0.9

click sidebar-toggle
wait 500

click nav-resources
wait_for nav-resources
wait 500

click nav-assistant
wait 700

# Variables
$id = read row-1
click $id

# Assertions
fill appbar-search "acme"
expect resources-table to_contain "Acme"
```

**Syntax:**
- One command per line.
- Whitespace-tolerant. Quote values with spaces (`"hello world"`). Backslash-escape inside quotes.
- `#` starts a comment. The `# speed: <n>` directive at the top sets cursor animation speed (default 1).
- `$name = <command>` assigns the command's return value to a variable.
- `$name` in args is interpolated at dispatch.
- `run other-script` includes another script (resolved as `/scripts/other-script.script`).

**API:**

```ts
import { parseScript, parseLine, stringifyScript } from "~/lib/command-parser"
import { runScript, runScriptText } from "~/lib/command-script"

const { options, commands } = parseScript(text)   // → JSON commands
const dsl = stringifyScript(commands, options)    // → text (round-trips)

await runScriptText(dsl)
await runScript("demo-tour")
```

## Scripts

Live in `public/scripts/*.script`. Two ways to invoke at runtime:

1. **Dialog** — appbar Play icon, or **⌘⇧P** keyboard shortcut. Lists known
   scripts + a paste-DSL textarea. See `app/components/scripts-dialog.tsx`.
2. **Console** — `runScript("demo-tour")` or `runScriptText("…")`.

Sub-scripts compose:

```
# in a parent script
run setup-fixtures
run actual-test
```

## Virtual cursor

`app/lib/virtual-cursor.ts`. A floating SVG cursor + ripple element appended
to `document.body`. The bus's `beforeCommand` hook moves the cursor to the
target element before each command and rips a ripple on `click`. Speed is
controlled by the script's `# speed:` header (default 1; lower = slower
animation).

`aria-hidden`/`role="presentation"` so screen readers ignore it. To run
silently (e.g. tests), pass `{ silent: true }` to `dispatch()` or `run()`.

## LLM integration

`app/lib/llm-tools.ts`. Two responsibilities:

### 1. Build the system prompt

`buildSystemPrompt({ path, includeActions })` returns a string with:

- A short preface ("You are the assistant in Comfy Cloud…")
- A compact DSL reference (~120 tokens — kept tight for small context windows)
- The current route
- A live snapshot of every visible `[data-action]` on screen, formatted as
  `- <id>: <label>` (skipping invisible/portal items)

The Assistant route rebuilds this on every send when **UI Control** is on,
so the model always sees the current page's actions.

### 2. Extract action blocks from streamed replies

The system prompt teaches the model to emit a fenced ` ```action ` block
when the user asks it to do something. After each assistant turn ends, the
Assistant route runs `runActionBlocks(message.content)` which:

1. Regex-extracts every ` ```action ... ``` ` block.
2. Feeds each through `runScriptText` → DSL parser → command bus.
3. Returns `{ ran, errors }` for the route to surface as a status pill
   ("Ran 2 actions").

The action-block fence renders as a small pill in the chat bubble (not raw
text) — see `app/components/assistant/message-body.tsx`.

### Format the model emits

```
I'll take you to resources.

​```action
navigate /resources
wait_for nav-resources
​```

Done — anything else?
```

The "rules" the system prompt teaches the model:

- Only emit a block when asked to *do* something. Questions and chitchat
  reply normally.
- Use only ids from the "Available actions" list.
- A short sentence + the block. Optional follow-up after.
- Never invent target ids.

Smaller models (≤7B) sometimes drift — the system prompt is explicit about
each rule, but you'll see occasional invented ids or extra prose. Bigger
models or Claude follow it near-perfectly.

### Token budget

`useChat` with these `reqExtras`:

- `system` — fresh per send via `buildSystemPrompt(...)`
- `messages` — pre-trimmed via `trimMessages(...)` to fit `contextTokens − sysTokens − responseBudget`
- `maxTokens: responseBudget` — caps the reply length

`contextTokens`, `responseBudget`, and `baseURL` come from `/settings`.
Default 9000 / 512. The header bar shows a live `<used> / <total>` badge
that turns amber when getting close.

## Producer 4: WebSocket (optional, unwired)

`app/lib/command-ws.ts`. A reconnecting `WebSocket` listener that accepts
three message shapes:

```json
{ "id": "abc", "command": { "type": "click", "target": "save-button" } }
{ "id": "abc", "script":  [ {"type":"navigate","path":"/x"}, … ] }
{ "id": "abc", "dsl":     "navigate /x\nclick save-button" }
```

Replies with `{ id, ok: true }` or `{ id, ok: false, error: "…" }`.

Not wired up by default. To enable for a session:

```ts
import { connectCommandSocket } from "~/lib/command-ws"
const sock = connectCommandSocket("ws://localhost:9229/ui")
// ... later ...
sock.close()
```

The intended use is screen-share / observer / CI scenarios where an external
process drives the UI. **Origin checks and an opt-in toggle in /settings are
sketched but not built** — don't auto-connect in production.

## Safety (sketch, not built)

When the UI gains destructive surfaces, mark them with
`data-action-danger`:

```tsx
<Button data-action="delete-account" data-action-danger>Delete</Button>
```

The bus's `beforeCommand` hook can refuse danger-marked targets unless the
caller passes a confirmation token. This is **not implemented yet** — flag
and design when you have a real destructive action to gate.

## Files at a glance

| File | Role |
|---|---|
| `app/lib/command-bus.ts` | JSON layer, dispatch, handlers, vars, history, `listActions`, `readState` |
| `app/lib/command-parser.ts` | DSL ↔ JSON |
| `app/lib/command-script.ts` | Script runner — load `/scripts/*.script`, run with cursor speed |
| `app/lib/virtual-cursor.ts` | Visible cursor + ripple |
| `app/lib/command-provider.tsx` | React glue: registers `navigate`, mounts cursor, exposes `window.*` |
| `app/lib/command-ws.ts` | Optional WebSocket producer |
| `app/lib/llm-tools.ts` | `buildSystemPrompt`, `extractActionBlocks`, `runActionBlocks`, token utils |
| `app/lib/llm-settings.ts` | Persisted base URL / context / response cap |
| `app/components/scripts-dialog.tsx` | ⌘⇧P script runner UI |
| `app/components/assistant/message-body.tsx` | Markdown bubble + action-block pill |
| `public/scripts/demo-*.script` | Examples |

## Quick recipes

**Make a new component scriptable** — add `data-action="<id>"`. Done.

**Test a flow** — write a `.script` file with `expect` assertions:

```
navigate /settings
fill settings-base-url "http://localhost:1234/v1"
click settings-test
wait 1500
expect settings-test to_contain "models available"
```

Run it via the dialog or `runScript("…")`.

**Drive a custom widget** — register a handler:

```ts
commandBus.register("highlight", async (cmd) => {
  /* ...your logic... */
})
```

Then dispatch `{ type: "highlight", target: "…" }` from anywhere.

**Send the LLM a different system prompt** — `buildSystemPrompt` accepts a
`preface` override:

```ts
buildSystemPrompt({
  preface: "You are the support agent for Comfy Cloud's billing team.",
  path: window.location.pathname,
})
```