diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..19cad89 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,211 @@ +# Research Workbench + +A general-purpose AI research tool: chat-driven, browser-equipped, artifact-generating. + +## What It Is + +A web UI backed by amplifierd that gives you a persistent AI research assistant with a live browser. You chat, it researches. It opens tabs, reads pages, extracts data, generates reports. You can take over the browser anytime. Every research session produces artifacts (guides, comparisons, reports) that persist. + +## Layout + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ ┌──────────┐ ┌───────────────────────┐ ┌──────────────────────────┐ │ +│ │ Sessions │ │ │ │ │ │ +│ │ │ │ Chat │ │ Browser (noVNC) │ │ +│ │ > Cars │ │ │ │ ── or ── │ │ +│ │ Jobs │ │ Messages stream here │ │ Artifacts │ │ +│ │ Travel │ │ Tool calls inline │ │ (guides, tables, docs) │ │ +│ │ │ │ A2UI controls inline │ │ │ │ +│ │ │ │ │ │ [Browser] [Artifacts] │ │ +│ │ + New │ │ ┌─────────────────┐ │ │ tabs to swap view │ │ +│ │ │ │ │ Type message... │ │ │ │ │ +│ └──────────┘ └───────────────────────┘ └──────────────────────────┘ │ +│ sidebar center right panel │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +- **Left sidebar**: session list. Each session is an independent research topic backed by an amplifierd session. Click to switch. "+ New" creates a new session. +- **Center**: chat. Full conversational interface. Messages stream via SSE from amplifierd. Tool calls (browser actions, searches) render inline as compact cards. A2UI blocks (filters, forms, toggles) are generated dynamically by the AI and rendered inline. User types at the bottom. +- **Right panel**: swappable between two views: + - **Browser**: noVNC iframe showing the live Playwright browser. AI navigates, user can take over. + - **Artifacts**: generated documents (guides, comparison tables, reports). Each artifact is a rendered markdown/HTML document. Multiple artifacts per session, tabbed. + +## Architecture + +``` +Web Frontend (static HTML/JS) + │ + │ HTTP + SSE + │ + ▼ +amplifierd (localhost:8410) + │ ── persistent Amplifier sessions + │ ── bundles, tools, hooks, providers + │ ── SSE event streaming + │ ── session management (create, resume, fork) + │ + │ tool calls + │ + ├──▶ playwright-cli (@playwright/cli) + │ ── token-efficient browser control + │ ── compact accessibility tree snapshots with refs + │ ── named sessions with persistent tabs + │ ── "playwright-cli -s=research1 open https://..." + │ ── "playwright-cli -s=research1 snapshot -ic" + │ ── "playwright-cli -s=research1 click @e5" + │ + ├──▶ other Amplifier tools (filesystem, bash, web, etc.) + │ + └──▶ artifact generation (write markdown/HTML to artifacts store) + +noVNC (port 6080) + │ ── Xvfb virtual display + │ ── x11vnc VNC server + │ ── websockify for browser access + │ ── shows the SAME browser playwright-cli controls + │ ── user can interact directly via VNC at any time +``` + +## Key Design Decisions + +### 1. amplifierd as backend (not raw Anthropic API) + +amplifierd gives us: +- Real Amplifier sessions with all bundles, tools, hooks +- Agent delegation (spawn sub-agents for parallel research) +- Session persistence (resume research later) +- SSE streaming (real-time chat updates) +- Tool approval gates (human-in-the-loop for destructive actions) +- Context management (long sessions with compaction) + +### 2. playwright-cli for browser control (not playwright-mcp) + +playwright-mcp is token-heavy -- it loads the full MCP tool schema and verbose DOM trees into context. playwright-cli is: +- CLI-invoked: `playwright-cli open`, `playwright-cli click @e3` +- Compact snapshots: refs like @e1, @e2 instead of full accessibility tree (~200 tokens vs 3000+) +- Named sessions: `playwright-cli -s=session1` keeps tabs open across commands +- Exposed as a regular Amplifier tool (tool-bash calls playwright-cli) + +### 3. Persistent tabs (not open/close) + +Each `playwright-cli -s=` command operates on the same browser session. New sites open in new tabs. The browser stays running on the Xvfb display. The user can see ALL open tabs in the noVNC view and take over any of them. + +### 4. A2UI (AI-to-UI) + +The AI can generate inline UI controls in the chat: +- Filter toggles: `{"type": "filters", "options": [{"label": "Hybrid", "active": true}, ...]}` +- Forms: `{"type": "form", "fields": [{"name": "max_price", "type": "number", "value": 10000}]}` +- Tables: `{"type": "table", "headers": [...], "rows": [...]}` +- Cards: `{"type": "cards", "items": [{"title": "...", "image": "...", "price": "..."}]}` + +These are rendered by the frontend from structured JSON blocks in the assistant's response. The AI chooses when to generate them based on context. + +### 5. Artifacts + +Generated documents that persist per-session: +- Buying guides, comparison tables, research reports +- Stored as markdown files in `artifacts//` +- Rendered in the right panel with tabs +- Can be exported, shared, or printed + +### 6. Multiple sessions + +Each research topic gets its own amplifierd session: +- Independent context, history, and browser state +- Sessions are listed in the sidebar +- Click to switch (loads that session's chat + artifacts + browser state) +- Sessions persist across page reloads (amplifierd manages state) + +## Services + +| Hostname | Port | Service | +|----------|------|---------| +| browser.ampbox.io | 8080 | Web frontend (FastAPI serves static + proxies to amplifierd) | +| vnc.ampbox.io | 6080 | noVNC live browser view | +| (internal) | 8410 | amplifierd daemon | + +## Browser Control Flow + +``` +User: "Find me hybrid cars under 10K near Woodinville" + │ + ▼ (amplifierd session) +AI: I'll search for hybrids in your area. + │ + ├── tool_call: bash("playwright-cli -s=cars open https://seattle.craigslist.org/search/cta?...") + │ → user sees browser navigate in noVNC + │ + ├── tool_call: bash("playwright-cli -s=cars snapshot -ic") + │ → AI gets compact element list with refs + │ + ├── tool_call: bash("playwright-cli -s=cars click @e15") // click a listing + │ → user sees browser click in noVNC + │ + ├── A2UI: renders filter toggles inline in chat + │ → user clicks "Toyota only" toggle + │ → frontend sends user message with filter state + │ + ├── tool_call: bash("playwright-cli -s=cars open https://cargurus.com/...") // new tab + │ → opens in a new tab, user sees both tabs in noVNC + │ + └── artifact: generates "Car Buying Guide - Hybrids Under $10K" + → renders in the right panel Artifacts tab +``` + +## Tech Stack + +| Component | Technology | +|-----------|-----------| +| Frontend | Vanilla HTML/JS/CSS (no framework) | +| Backend proxy | FastAPI (thin, proxies to amplifierd) | +| AI runtime | amplifierd (Amplifier sessions) | +| Browser control | playwright-cli (token-efficient) | +| Browser display | Xvfb + x11vnc + noVNC | +| Tunnel | cloudflared (browser.ampbox.io, vnc.ampbox.io) | +| Container | Docker (Xvfb + noVNC + playwright-cli + FastAPI) | + +## File Structure + +``` +research-workbench/ +├── Dockerfile +├── entrypoint.sh +├── app.py # FastAPI: serves frontend, proxies amplifierd API +├── static/ +│ └── index.html # Single-page app +├── artifacts/ # Generated documents per session +│ └── / +│ ├── guide.md +│ └── comparison.md +├── bundle/ # Amplifier bundle for research agent +│ ├── bundle.md # research-workbench bundle definition +│ └── agents/ +│ └── researcher.md # research agent with browser + artifact tools +└── README.md +``` + +## Bundle: research-workbench + +The Amplifier bundle that powers the research agent: + +```yaml +bundle: + name: research-workbench + version: 0.1.0 + +tools: + - module: tool-bash # for playwright-cli commands + - module: tool-filesystem # for artifact generation + - module: tool-web-fetch # for quick HTTP fetches + +agents: + include: + - researcher +``` + +The `researcher` agent knows how to: +- Use playwright-cli for browser control +- Generate A2UI blocks for interactive controls +- Create artifacts (guides, reports, tables) +- Conduct multi-step research with persistent browser tabs