Files
research-workbench/docs/plans/2026-05-26-research-workbench-phase3.md
Ken 4656af010e docs: implementation plans for research workbench (3 phases, 30 tasks)
Phase 1: Infrastructure (8 tasks) - Dockerfile, amplifierd, FastAPI, auth, AG-UI adapter
Phase 2: Frontend (10 tasks) - React/TS/Vite, Tabler, CopilotKit headless, SSE streaming
Phase 3: Bundle + Browser + Artifacts (12 tasks) - Amplifier bundle, researcher agent,
  AI-generated visual artifacts (Claude builds visuals style), noVNC browser panel
2026-05-26 18:01:47 +00:00

1586 lines
57 KiB
Markdown

# Research Workbench — Phase 3: Bundle + Browser + Artifacts
> **Execution:** Use the subagent-driven-development workflow to implement this plan.
**Goal:** Create the Amplifier bundle (researcher agent), wire up the browser panel (noVNC iframe), implement the artifacts panel with markdown rendering, add A2UI block rendering for inline AI-generated UI controls, and build the visual artifact runtime for AI-generated interactive inline visualizations (Claude "builds visuals" style).
**Architecture:** The bundle follows the thin-root pattern: `bundle.md` includes foundation (which brings tool-bash, tool-filesystem, tool-web-fetch) and declares the researcher agent. The agent's body contains playwright-cli command reference, A2UI output format spec, artifact conventions, and visual artifact generation instructions. A single universal runtime (`bundle/apps/runtime.html`) executes arbitrary AI-generated code (React/JSX, Chart.js, Leaflet, plain HTML) in a sandboxed iframe. The AI generates visualization code on the fly during conversation — not from fixed templates. The browser panel embeds noVNC via iframe. The artifacts panel lists and renders markdown files written by the agent.
**Tech Stack:** Amplifier bundle (markdown + YAML), React components (Tabler), react-markdown, noVNC iframe, Chart.js, Leaflet.js, React 19/Babel standalone (in-browser JSX), MCP Apps (SEP-1865)
**Depends on:** Phase 1 (backend) and Phase 2 (frontend shell) complete
---
### Task 1: Bundle root (bundle.md)
The thin root bundle. It includes foundation (which provides tool-bash, tool-filesystem, tool-web-fetch) and declares the researcher agent. Do NOT redeclare foundation tools — that's the canonical anti-pattern.
**Files:**
- Create: `bundle/bundle.md`
**Step 1: Create bundle.md**
Create `bundle/bundle.md`:
```markdown
---
bundle:
name: research-workbench
version: 0.1.0
description: General-purpose AI research tool with browser control and artifact generation
includes:
- bundle: git+https://github.com/microsoft/amplifier-foundation@main
- bundle: research-workbench:behaviors/research-workbench
---
# Research Workbench
A research-first Amplifier bundle. Provides a researcher agent with browser
control (via playwright-cli), artifact generation, and A2UI inline controls.
## What's Included
**From foundation:** tool-bash, tool-filesystem, tool-web-fetch, and all standard hooks.
**From this bundle:**
- `researcher` agent — browser-equipped research assistant
- Research workbench behavior — wires the researcher agent with context
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add bundle/bundle.md && git commit -m "feat: thin root bundle including foundation"
```
---
### Task 2: Research workbench behavior
**Files:**
- Create: `bundle/behaviors/research-workbench.yaml`
**Step 1: Create the behavior file**
Create `bundle/behaviors/research-workbench.yaml`:
```yaml
# research-workbench behavior
# Wires the researcher agent and research context into sessions.
behavior:
name: research-workbench
description: Browser-equipped research assistant with artifact generation
agents:
include:
- researcher
context:
include:
- research-workbench:context/researcher-instructions
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add bundle/behaviors/research-workbench.yaml && git commit -m "feat: research-workbench behavior wiring researcher agent"
```
---
### Task 3: Researcher agent definition
The researcher agent is the core of the system. It knows how to use playwright-cli for browser control, generate A2UI blocks for interactive controls, and create artifacts.
**Files:**
- Create: `bundle/agents/researcher.md`
**Step 1: Create the researcher agent**
Create `bundle/agents/researcher.md`:
```markdown
---
meta:
name: researcher
description: |
WHY: Users need a hands-free research assistant that can browse the web,
extract data, compare options, and generate polished deliverables.
WHEN: Delegated to for any research task — product comparisons, market
research, technical investigations, travel planning, buying guides.
WHAT: Conducts multi-step web research using a real browser (playwright-cli),
generates interactive inline UI controls (A2UI), and produces artifacts
(guides, reports, comparison tables) as markdown files.
HOW: Opens browser tabs via playwright-cli, takes compact snapshots,
clicks elements, fills forms. Generates A2UI JSON blocks inline when the
user would benefit from filters, forms, or data tables. Writes artifacts
to the artifacts directory for persistent deliverables.
Examples of delegation:
- "Research hybrid cars under $10K near Woodinville WA"
- "Compare the top 5 project management tools for small teams"
- "Find flights from Seattle to Tokyo in October under $800"
- "Investigate the best mechanical keyboards for programming"
model_role:
- research
- general
---
# Researcher Agent
You are a research assistant with a live web browser. You help users
investigate topics, compare options, and produce polished deliverables.
## Browser Control (playwright-cli)
You control a Chromium browser via `playwright-cli`. The browser is visible
to the user in real-time through a VNC panel.
### Commands
All commands use `playwright-cli -s=research` (the `-s` flag keeps a persistent
browser session across commands).
| Action | Command |
|--------|---------|
| Open a URL | `playwright-cli -s=research open https://example.com` |
| Take a snapshot | `playwright-cli -s=research snapshot` |
| Compact snapshot | `playwright-cli -s=research snapshot -ic` |
| Click an element | `playwright-cli -s=research click @e5` |
| Type text | `playwright-cli -s=research type @e3 "search query"` |
| Press a key | `playwright-cli -s=research press Enter` |
| Scroll down | `playwright-cli -s=research scroll down` |
| Go back | `playwright-cli -s=research back` |
| New tab | `playwright-cli -s=research open https://other-site.com` |
| List tabs | `playwright-cli -s=research tabs` |
| Switch tab | `playwright-cli -s=research tab 2` |
### Snapshot Refs
Snapshots return an accessibility tree with element refs like `@e1`, `@e2`, etc.
Use these refs for click/type actions. Always take a snapshot after navigation
to see the page state.
### Best Practices
1. Always take a compact snapshot (`-ic`) after opening a URL or clicking
2. Use refs from the most recent snapshot — they change after navigation
3. Open new sites in new tabs (both URLs will be visible in the VNC panel)
4. The user can see and interact with the browser via VNC at any time
## A2UI (AI-to-UI) Blocks
You can generate interactive UI controls inline in your messages. Wrap them
in a JSON code block with the `a2ui` language tag. The frontend renders them
as Tabler components.
@mention bundle/context/a2ui-format.md
## Artifacts
When you produce a substantial deliverable (guide, report, comparison table),
save it as a markdown file in the artifacts directory.
@mention bundle/context/artifact-conventions.md
## Inline Visuals (Claude-style)
You can generate interactive visualizations inline in your messages. When your
research would benefit from a visual (comparison chart, data table, map, dashboard,
decision matrix), write the code directly as a `visual` code block. The frontend
renders it live in a sandboxed iframe.
@mention bundle/context/visual-artifacts.md
Be proactive: if you're comparing 5 products, generate a comparison table.
If you're showing locations, generate a map. If you're showing trends, generate
a chart. Don't wait to be asked.
## Research Workflow
1. **Understand the request** — clarify scope, constraints, preferences
2. **Plan the research** — identify sites to check, data to gather
3. **Browse and extract** — open sites, take snapshots, extract data
4. **Present findings** — use A2UI blocks for simple controls, `visual` blocks for rich interactive visualizations
5. **Generate artifacts** — write polished guides/reports as markdown files
6. **Iterate** — refine based on user feedback, apply filters, dig deeper
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add bundle/agents/researcher.md && git commit -m "feat: researcher agent with playwright-cli, A2UI, inline visuals, artifact conventions"
```
---
### Task 4: Context files (A2UI format, artifact conventions, instructions)
**Files:**
- Create: `bundle/context/a2ui-format.md`
- Create: `bundle/context/artifact-conventions.md`
- Create: `bundle/context/researcher-instructions.md`
**Step 1: Create a2ui-format.md**
Create `bundle/context/a2ui-format.md`:
````markdown
# A2UI Block Format
Generate interactive UI by emitting a fenced code block with language `a2ui`
containing a JSON object. The frontend parses these and renders Tabler
components.
## Supported Block Types
### Filters
Toggle buttons for filtering results. When the user clicks a filter, the
frontend sends a message with the updated filter state.
```a2ui
{
"type": "filters",
"options": [
{"label": "Toyota", "active": true},
{"label": "Honda", "active": true},
{"label": "Hyundai", "active": false},
{"label": "Under $5K", "active": false}
]
}
```
### Table
Data table with headers and rows. Good for comparisons.
```a2ui
{
"type": "table",
"headers": ["Model", "Year", "Price", "Miles", "Source"],
"rows": [
["Prius", "2014", "$6,500", "89K", "Craigslist"],
["Camry Hybrid", "2012", "$7,200", "102K", "CarGurus"]
]
}
```
### Cards
Card grid for items with images (listings, products, places).
```a2ui
{
"type": "cards",
"items": [
{
"title": "2014 Toyota Prius",
"description": "Clean title, one owner, 89K miles",
"price": "$6,500",
"url": "https://example.com/listing/123"
}
]
}
```
### Form
Input form for structured user preferences.
```a2ui
{
"type": "form",
"fields": [
{"name": "max_price", "type": "number", "label": "Max Price", "value": 10000},
{"name": "radius", "type": "number", "label": "Search Radius (mi)", "value": 50},
{"name": "fuel_type", "type": "select", "label": "Fuel", "options": ["hybrid", "electric", "any"]}
],
"submitLabel": "Update Search"
}
```
## Rules
- Always use the `a2ui` language tag on the code fence
- The JSON must be a single object with a `type` field
- Generate A2UI blocks when the user would benefit from interactive controls
- Don't overuse — plain text is fine for simple responses
````
**Step 2: Create artifact-conventions.md**
Create `bundle/context/artifact-conventions.md`:
```markdown
# Artifact Conventions
Artifacts are polished deliverables saved as markdown files. They persist
across the session and are viewable in the Artifacts panel.
## Writing Artifacts
Use tool-filesystem to write files to the artifacts directory:
```
write_file("artifacts/<session_id>/<name>.md", content)
```
The `<session_id>` matches the current session. Ask the system for the
session ID if you don't know it, or use the working directory path.
## File Naming
- Use kebab-case: `buying-guide.md`, `comparison-table.md`
- Be descriptive: `hybrid-cars-under-10k-guide.md`
- Always use `.md` extension
## Content Format
- Start with an H1 title
- Include a date and scope summary at the top
- Use markdown tables for comparisons
- Use bullet points for recommendations
- Include source URLs where applicable
- End with a "Next Steps" or "Summary" section
## When to Create Artifacts
Create an artifact when:
- The user asks for a guide, report, or summary
- You've gathered enough data for a comparison table
- The research produces actionable recommendations
- The user would benefit from a printable/shareable document
Don't create artifacts for:
- Simple answers or clarifications
- Intermediate research steps
- Status updates
```
**Step 3: Create researcher-instructions.md (thin awareness pointer)**
Create `bundle/context/researcher-instructions.md`:
```markdown
# Research Workbench Context
This session runs inside the Research Workbench. You have access to:
- **Browser**: A live Chromium browser controlled via `playwright-cli -s=research`.
The user can see the browser in the VNC panel and take over at any time.
- **Artifacts**: Write research deliverables to `artifacts/<session_id>/`.
They appear in the Artifacts panel.
- **A2UI**: Generate inline interactive controls using `a2ui` code blocks.
- **Inline Visuals**: Generate interactive charts, tables, maps, and dashboards
using `visual` code blocks. See the researcher agent for full details.
See the researcher agent definition for full command reference.
```
**Step 4: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add bundle/context/ && git commit -m "feat: context files for A2UI format, artifact conventions, researcher instructions"
```
---
### Task 5: Visual artifact runtime (AI-generated inline visuals)
A single universal runtime that executes AI-generated visualization code in a sandboxed iframe. Instead of fixed templates, the AI generates arbitrary React/HTML/JS code during conversation (Claude "builds visuals" style), and this runtime renders it live. Pre-loads common visualization libraries so the AI can use them immediately.
**Files:**
- Create: `bundle/apps/runtime.html`
**Step 1: Create runtime.html**
Create `bundle/apps/runtime.html`:
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- Pre-loaded libraries available to all AI-generated visuals -->
<link href="https://cdn.jsdelivr.net/npm/@tabler/core@1.0.0-beta20/dist/css/tabler.min.css" rel="stylesheet">
<link href="https://cdn.jsdelivr.net/npm/leaflet@1.9/dist/leaflet.css" rel="stylesheet">
<script src="https://cdn.jsdelivr.net/npm/chart.js@4"></script>
<script src="https://cdn.jsdelivr.net/npm/leaflet@1.9/dist/leaflet.js"></script>
<script src="https://cdn.jsdelivr.net/npm/react@19/umd/react.production.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/react-dom@19/umd/react-dom.production.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@babel/standalone@7/babel.min.js"></script>
<style>
body {
background: transparent;
margin: 0;
padding: 16px;
color: #e0e0e0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
}
.theme-dark { color-scheme: dark; }
.visual-error {
color: #f7607a;
background: #2a1a1e;
border: 1px solid #4a2a3a;
border-radius: 8px;
padding: 1rem;
font-family: monospace;
font-size: 0.8rem;
white-space: pre-wrap;
}
</style>
</head>
<body class="theme-dark">
<div id="root"></div>
<script>
// --- MCP Apps JSON-RPC 2.0 bridge (pre-wired) ---
const mcpBridge = {
_pending: new Map(),
sendToHost(msg) {
window.parent.postMessage(msg, "*");
},
callTool(name, args) {
const id = crypto.randomUUID();
this.sendToHost({
jsonrpc: "2.0", id,
method: "tools/call",
params: { name, arguments: args },
});
return new Promise((resolve) => this._pending.set(id, resolve));
},
sendMessage(content) {
this.sendToHost({
jsonrpc: "2.0",
method: "ui/sendMessage",
params: { role: "user", content },
});
},
resize(height) {
this.sendToHost({
jsonrpc: "2.0",
method: "ui/resize",
params: { height: Math.min(height, 800) },
});
},
};
// --- Visual execution engine ---
function executeVisual(code, data) {
const root = document.getElementById("root");
try {
// Make data and bridge available to the generated code
window.__VISUAL_DATA__ = data || {};
window.__MCP__ = mcpBridge;
// Detect if the code contains JSX (look for < followed by a capital letter
// or common HTML tags used in React-like syntax)
const hasJSX = /<[A-Z]/.test(code) || /React\.createElement/.test(code);
const hasBabelType = /type\s*=\s*["']text\/babel["']/.test(code);
if (hasJSX || hasBabelType) {
// --- React/JSX path: compile with Babel, render to #root ---
const compiled = Babel.transform(code, {
presets: ["react"],
filename: "visual.jsx",
}).code;
// Wrap in a function that has access to React, ReactDOM, data, and bridge
const fn = new Function(
"React", "ReactDOM", "data", "mcp", "root",
compiled + "\n\n" +
"// Auto-render: if code defined a default export or App component\n" +
"if (typeof App !== 'undefined') {\n" +
" ReactDOM.createRoot(root).render(React.createElement(App, { data, mcp }));\n" +
"}\n"
);
fn(React, ReactDOM, data, mcpBridge, root);
} else if (/<[a-z][\s\S]*>/i.test(code)) {
// --- HTML path: inject directly ---
root.innerHTML = code;
// Execute any <script> tags in the injected HTML
root.querySelectorAll("script").forEach((oldScript) => {
const newScript = document.createElement("script");
if (oldScript.src) {
newScript.src = oldScript.src;
} else {
// Wrap script body so it has access to data and bridge
newScript.textContent =
"(function(data, mcp) {\n" +
oldScript.textContent +
"\n})(window.__VISUAL_DATA__, window.__MCP__);";
}
oldScript.parentNode.replaceChild(newScript, oldScript);
});
} else {
// --- Plain JS path: execute as a script that manipulates #root ---
const fn = new Function("data", "mcp", "root", "Chart", "L", code);
fn(data, mcpBridge, root, window.Chart, window.L);
}
// Auto-resize after rendering
requestAnimationFrame(() => {
const height = Math.max(root.scrollHeight + 32, 100);
mcpBridge.resize(height);
});
} catch (err) {
root.innerHTML =
'<div class="visual-error">Visual render error:\n' +
err.message.replace(/</g, "&lt;") +
"</div>";
mcpBridge.resize(120);
}
}
// --- Listen for messages from the host ---
window.addEventListener("message", (e) => {
const msg = e.data;
if (!msg || typeof msg !== "object") return;
// Response to a tools/call we made
if (msg.id && mcpBridge._pending.has(msg.id)) {
mcpBridge._pending.get(msg.id)(msg.result);
mcpBridge._pending.delete(msg.id);
return;
}
// Host sends visual code to render
if (msg.method === "render") {
executeVisual(msg.params?.code || "", msg.params?.data || {});
return;
}
// Legacy: tool/input for backwards compatibility with MCP App protocol
if (msg.method === "tool/input") {
// If params has a `code` field, treat as a visual render
if (msg.params?.code) {
executeVisual(msg.params.code, msg.params.data || {});
}
}
});
// Tell the host we're ready
mcpBridge.sendToHost({
jsonrpc: "2.0",
method: "ui/initialize",
params: { runtime: "visual-artifact", version: "1.0.0" },
});
</script>
</body>
</html>
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add bundle/apps/runtime.html && git commit -m "feat: visual artifact runtime for AI-generated inline visualizations"
```
---
### Task 6: Visual artifacts agent context file
This context file teaches the researcher agent HOW to generate inline visuals — Claude "builds visuals" style. The agent generates arbitrary React/HTML/JS code on the fly, not from fixed templates. The context includes when to generate visuals, what libraries are available, the output format, 5 complete working example patterns, and constraints.
**Files:**
- Create: `bundle/context/visual-artifacts.md`
**Step 1: Create the visual artifacts context file**
Create `bundle/context/visual-artifacts.md`:
````markdown
# Inline Visual Artifacts
You can generate interactive visualizations inline in your messages by writing
code in a `visual` fenced code block. The frontend renders the code live in
a sandboxed iframe with pre-loaded libraries.
## When to Generate a Visual
Be proactive. Generate a visual when:
- **Comparing options**: product comparison tables, feature matrices, pros/cons
- **Showing trends**: price history, ratings over time, market data
- **Displaying locations**: dealer maps, store finders, event venues
- **Scoring/ranking**: decision matrices, weighted scoring tables
- **Budget breakdowns**: cost comparisons, price distributions
- **Dashboards**: multi-metric summaries with charts + stats + tables
Don't wait to be asked. If you have structured data that would benefit from
a visual, generate one.
## Available Libraries (pre-loaded in the runtime)
These are loaded and ready to use — no imports needed:
| Library | Global | Version | Use for |
|---------|--------|---------|---------|
| **Tabler CSS** | CSS classes | 1.0.0-beta20 | Cards, tables, badges, buttons, forms, grid, dark theme |
| **Chart.js** | `Chart` | 4.x | Bar, line, pie, radar, doughnut, scatter, polar area charts |
| **Leaflet.js** | `L` | 1.9 | Maps with markers, popups, tile layers (CARTO dark tiles) |
| **React** | `React` | 19 | Interactive components with state |
| **ReactDOM** | `ReactDOM` | 19 | Rendering React components |
| **Babel** | (automatic) | 7.x | JSX compilation (used automatically when JSX detected) |
## Output Format
Write the visual as a fenced code block with the `visual` language tag,
containing a JSON object with `title`, `code`, and optionally `data` and `height`:
~~~
```visual
{
"title": "Price Comparison",
"code": "<your complete HTML, JSX, or JS code here>",
"data": { "items": [...] },
"height": 400
}
```
~~~
### Fields
| Field | Required | Description |
|-------|----------|-------------|
| `title` | Yes | Display title shown above the visual |
| `code` | Yes | The complete code to execute (HTML, JSX, or plain JS) |
| `data` | No | Structured data object passed to the code as `data` |
| `height` | No | Initial iframe height in pixels (default: 300, max: 800) |
### Code Formats
The runtime auto-detects the code format:
1. **React/JSX** (contains `<ComponentName` or `React.createElement`): Compiled with Babel, rendered to `#root`. Define an `App` component and it auto-renders.
2. **HTML** (contains `<div`, `<table`, etc.): Injected directly into `#root`. Script tags execute with `data` and `mcp` in scope.
3. **Plain JS** (anything else): Executed as a function with `data`, `mcp`, `root`, `Chart`, `L` as parameters.
### Available in Code Scope
| Variable | Type | Description |
|----------|------|-------------|
| `data` | Object | The `data` field from the visual JSON block |
| `mcp` | Object | MCP bridge: `mcp.sendMessage(text)`, `mcp.callTool(name, args)`, `mcp.resize(height)` |
| `root` | Element | The `#root` DOM element (for plain JS path) |
| `Chart` | Class | Chart.js constructor (for plain JS path) |
| `L` | Object | Leaflet.js namespace (for plain JS path) |
## Example Patterns
### 1. Comparison Table (HTML + Tabler CSS)
```visual
{
"title": "Hybrid Cars Under $10K",
"code": "<div class=\"card\"><div class=\"table-responsive\"><table class=\"table table-hover table-striped\"><thead><tr><th>Model</th><th>Year</th><th>Price</th><th>Miles</th><th>MPG</th><th>Source</th></tr></thead><tbody id=\"rows\"></tbody></table></div></div><script>const rows = data.items.map(item => `<tr><td><strong>${item.model}</strong></td><td>${item.year}</td><td style=\"color:#7af7a2;font-weight:600\">${item.price}</td><td>${item.miles}</td><td>${item.mpg}</td><td><a href=\"${item.url}\" style=\"color:#7aa2f7\" target=\"_blank\">${item.source}</a></td></tr>`).join('');document.getElementById('rows').innerHTML = rows;</script>",
"data": {
"items": [
{"model": "Toyota Prius", "year": 2015, "price": "$7,800", "miles": "82K", "mpg": "51", "source": "CarGurus", "url": "#"},
{"model": "Honda Civic Hybrid", "year": 2014, "price": "$6,200", "miles": "95K", "mpg": "44", "source": "Craigslist", "url": "#"},
{"model": "Toyota Camry Hybrid", "year": 2013, "price": "$8,500", "miles": "78K", "mpg": "41", "source": "AutoTrader", "url": "#"},
{"model": "Ford Fusion Hybrid", "year": 2014, "price": "$5,900", "miles": "110K", "mpg": "42", "source": "Cars.com", "url": "#"}
]
},
"height": 280
}
```
### 2. Bar Chart (Plain JS + Chart.js)
```visual
{
"title": "Price Comparison by Model",
"code": "const canvas = document.createElement('canvas'); root.appendChild(canvas); new Chart(canvas.getContext('2d'), { type: 'bar', data: { labels: data.labels, datasets: [{ label: data.datasetLabel, data: data.values, backgroundColor: 'rgba(122,162,247,0.6)', borderColor: 'rgba(122,162,247,1)', borderWidth: 1 }] }, options: { responsive: true, plugins: { legend: { labels: { color: '#c0c0d0' } }, title: { display: true, text: data.title, color: '#e0e0e0' } }, scales: { x: { ticks: { color: '#8888aa' }, grid: { color: '#1e2a4a' } }, y: { ticks: { color: '#8888aa' }, grid: { color: '#1e2a4a' } } }, onClick: (_, els) => { if (els.length) mcp.sendMessage('Tell me more about ' + data.labels[els[0].index]); } } });",
"data": {
"title": "Asking Price ($)",
"datasetLabel": "Price",
"labels": ["Prius", "Civic Hybrid", "Camry Hybrid", "Fusion Hybrid", "Insight"],
"values": [7800, 6200, 8500, 5900, 4800]
},
"height": 380
}
```
### 3. Location Map (Plain JS + Leaflet)
```visual
{
"title": "Dealers Near Woodinville, WA",
"code": "root.style.height = '340px'; const map = L.map(root).setView(data.center, data.zoom); L.tileLayer('https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}{r}.png', { attribution: 'CARTO', maxZoom: 18 }).addTo(map); data.markers.forEach(m => { const marker = L.marker([m.lat, m.lng]).addTo(map); marker.bindPopup(`<b>${m.title}</b><br>${m.description || ''}<br><span style='color:#7af7a2'>${m.price || ''}</span>`); marker.on('click', () => mcp.sendMessage('Tell me about ' + m.title)); }); if (data.markers.length > 1) map.fitBounds(data.markers.map(m => [m.lat, m.lng]), { padding: [30, 30] });",
"data": {
"center": [47.76, -122.16],
"zoom": 11,
"markers": [
{"lat": 47.758, "lng": -122.164, "title": "Toyota of Kirkland", "description": "Certified pre-owned", "price": "$7,800"},
{"lat": 47.81, "lng": -122.20, "title": "Honda of Bothell", "description": "Used inventory", "price": "$6,200"},
{"lat": 47.68, "lng": -122.12, "title": "CarMax Bellevue", "description": "Multi-brand", "price": "Various"}
]
},
"height": 380
}
```
### 4. React Dashboard (JSX)
```visual
{
"title": "Research Summary Dashboard",
"code": "function StatCard({ label, value, color }) { return React.createElement('div', { className: 'card', style: { flex: 1, minWidth: 140 } }, React.createElement('div', { className: 'card-body', style: { padding: '1rem' } }, React.createElement('div', { style: { fontSize: '0.75rem', color: '#8888aa', textTransform: 'uppercase' } }, label), React.createElement('div', { style: { fontSize: '1.5rem', fontWeight: 700, color: color || '#e0e0e0' } }, value))); } function App({ data }) { const [sort, setSort] = React.useState('price'); const sorted = [...data.items].sort((a, b) => sort === 'price' ? a.priceNum - b.priceNum : a.miles - b.miles); return React.createElement('div', null, React.createElement('div', { style: { display: 'flex', gap: '0.5rem', marginBottom: '1rem' } }, React.createElement(StatCard, { label: 'Options Found', value: data.items.length, color: '#7aa2f7' }), React.createElement(StatCard, { label: 'Avg Price', value: '$' + Math.round(data.items.reduce((s, i) => s + i.priceNum, 0) / data.items.length).toLocaleString(), color: '#7af7a2' }), React.createElement(StatCard, { label: 'Price Range', value: '$' + Math.min(...data.items.map(i => i.priceNum)).toLocaleString() + ' - $' + Math.max(...data.items.map(i => i.priceNum)).toLocaleString() })), React.createElement('div', { style: { marginBottom: '0.5rem' } }, React.createElement('button', { onClick: () => setSort('price'), className: 'btn btn-sm ' + (sort === 'price' ? 'btn-primary' : 'btn-outline-secondary'), style: { marginRight: 4 } }, 'Sort by Price'), React.createElement('button', { onClick: () => setSort('miles'), className: 'btn btn-sm ' + (sort === 'miles' ? 'btn-primary' : 'btn-outline-secondary') }, 'Sort by Miles')), React.createElement('div', { className: 'table-responsive' }, React.createElement('table', { className: 'table table-hover' }, React.createElement('thead', null, React.createElement('tr', null, ['Model', 'Price', 'Miles', 'Source'].map(h => React.createElement('th', { key: h }, h)))), React.createElement('tbody', null, sorted.map((item, i) => React.createElement('tr', { key: i, onClick: () => mcp.sendMessage('Tell me more about ' + item.model), style: { cursor: 'pointer' } }, React.createElement('td', null, React.createElement('strong', null, item.model)), React.createElement('td', { style: { color: '#7af7a2' } }, item.price), React.createElement('td', null, item.milesStr), React.createElement('td', null, item.source))))))); }",
"data": {
"items": [
{"model": "2015 Prius", "price": "$7,800", "priceNum": 7800, "miles": 82000, "milesStr": "82K", "source": "CarGurus"},
{"model": "2014 Civic Hybrid", "price": "$6,200", "priceNum": 6200, "miles": 95000, "milesStr": "95K", "source": "Craigslist"},
{"model": "2013 Camry Hybrid", "price": "$8,500", "priceNum": 8500, "miles": 78000, "milesStr": "78K", "source": "AutoTrader"},
{"model": "2014 Fusion Hybrid", "price": "$5,900", "priceNum": 5900, "miles": 110000, "milesStr": "110K", "source": "Cars.com"}
]
},
"height": 450
}
```
### 5. Decision Matrix (HTML + interactive weight sliders)
```visual
{
"title": "Car Decision Matrix",
"code": "<style>.matrix-table th, .matrix-table td { padding: 0.4rem 0.6rem; text-align: center; } .matrix-table th { color: #8888aa; font-size: 0.75rem; text-transform: uppercase; } .score-cell { font-weight: 600; } .best { color: #7af7a2; } .weight-slider { width: 80px; accent-color: #7aa2f7; }</style><div class=\"card\"><div class=\"card-body\"><div id=\"weights-row\" style=\"display:flex;gap:1rem;margin-bottom:1rem;flex-wrap:wrap\"></div><div class=\"table-responsive\"><table class=\"table table-hover matrix-table\" id=\"matrix\"></table></div></div></div><script>const criteria = data.criteria; const options = data.options; let weights = criteria.map(c => c.weight); function render() { const total = weights.reduce((s,w) => s+w, 0) || 1; const norm = weights.map(w => w/total); let html = '<thead><tr><th>Option</th>'; criteria.forEach((c,i) => html += `<th>${c.name} (${Math.round(norm[i]*100)}%)</th>`); html += '<th>Score</th></tr></thead><tbody>'; const scores = options.map(opt => { let s = 0; opt.scores.forEach((v,i) => s += v * norm[i]); return s; }); const maxScore = Math.max(...scores); options.forEach((opt,oi) => { const isMax = scores[oi] === maxScore; html += `<tr><td style=\"text-align:left\"><strong>${opt.name}</strong></td>`; opt.scores.forEach(s => html += `<td class=\"score-cell\">${s}/10</td>`); html += `<td class=\"score-cell ${isMax ? 'best' : ''}\" style=\"font-size:1.1rem\">${scores[oi].toFixed(1)}</td></tr>`; }); html += '</tbody>'; document.getElementById('matrix').innerHTML = html; } const wr = document.getElementById('weights-row'); criteria.forEach((c,i) => { const label = document.createElement('label'); label.style.cssText = 'font-size:0.75rem;color:#aaa;display:flex;flex-direction:column;align-items:center'; label.textContent = c.name; const slider = document.createElement('input'); slider.type = 'range'; slider.min = 0; slider.max = 10; slider.value = weights[i]; slider.className = 'weight-slider'; slider.oninput = () => { weights[i] = +slider.value; render(); }; label.appendChild(slider); wr.appendChild(label); }); render();</script>",
"data": {
"criteria": [
{"name": "Price", "weight": 8},
{"name": "MPG", "weight": 7},
{"name": "Reliability", "weight": 9},
{"name": "Cargo", "weight": 4},
{"name": "Comfort", "weight": 5}
],
"options": [
{"name": "2015 Prius", "scores": [7, 10, 9, 5, 6]},
{"name": "2014 Civic Hybrid", "scores": [8, 8, 8, 6, 7]},
{"name": "2013 Camry Hybrid", "scores": [6, 7, 9, 8, 9]},
{"name": "2014 Fusion Hybrid", "scores": [9, 7, 6, 7, 8]}
]
},
"height": 400
}
```
## Constraints
- **Self-contained**: All code must work without external API calls or fetches.
Data comes from the `data` field or is hardcoded in the code.
- **Dark theme always**: Use Tabler's `.theme-dark` classes. Background is
transparent (inherits from runtime). Text should be light (#e0e0e0).
- **Responsive width**: Visuals fill the available width. Use percentage widths
and `max-width` where appropriate.
- **Max height**: Specify `height` in the JSON block (default 300, max 800).
The runtime auto-resizes based on content.
- **No external fetches**: Don't call APIs from within the visual. All data
must be passed via the `data` field.
- **Click-to-explore**: When users click interactive elements (chart bars,
table rows, map markers), call `mcp.sendMessage("Tell me more about X")`
to inject a follow-up question into the chat.
## When to Use Visuals vs A2UI
| Need | Use |
|------|-----|
| Toggle filters, simple form inputs | A2UI blocks |
| Static data table (< 5 rows) | A2UI table block |
| Interactive chart with click events | `visual` block |
| Sortable table with many rows | `visual` block |
| Side-by-side product comparison | `visual` block |
| Location map with markers | `visual` block |
| Multi-metric dashboard | `visual` block |
| Decision matrix with adjustable weights | `visual` block |
````
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add bundle/context/visual-artifacts.md && git commit -m "feat: visual artifacts context file with 5 example patterns for AI-generated visuals"
```
---
### Task 7: Verify bundle structure and visual artifact references
The researcher agent definition references `@mention bundle/context/visual-artifacts.md` (from Task 3). This task verifies the agent can be loaded, the bundle structure is valid, and all @mention references resolve.
**Files:**
- Verify: `bundle/agents/researcher.md` (updated in Task 3 with inline visuals section)
- Verify: `bundle/apps/runtime.html` (created in Task 5)
- Verify: `bundle/context/visual-artifacts.md` (created in Task 6)
**Step 1: Verify bundle directory structure**
Run:
```bash
cd /home/ken/workspace/research-workbench
find bundle/ -type f | sort
```
Expected output:
```
bundle/agents/researcher.md
bundle/apps/runtime.html
bundle/behaviors/research-workbench.yaml
bundle/bundle.md
bundle/context/a2ui-format.md
bundle/context/artifact-conventions.md
bundle/context/researcher-instructions.md
bundle/context/visual-artifacts.md
```
**Step 2: Verify all @mention references resolve**
Run:
```bash
cd /home/ken/workspace/research-workbench
# Check that all @mention targets exist
grep -rh '@mention' bundle/ | sed 's/.*@mention //' | while read ref; do
# Convert bundle reference to file path
path="bundle/$(echo "$ref" | sed 's|^bundle/||').md"
if [ -f "$path" ]; then
echo "OK: $path"
else
echo "MISSING: $path (from @mention $ref)"
fi
done
```
Expected: All references resolve to OK. Specifically:
- `bundle/context/a2ui-format.md` — OK
- `bundle/context/artifact-conventions.md` — OK
- `bundle/context/visual-artifacts.md` — OK
- `bundle/context/researcher-instructions.md` — OK (via behavior include)
**Step 3: Verify runtime.html loads the expected libraries**
Run:
```bash
cd /home/ken/workspace/research-workbench
# Quick sanity check that runtime.html references all expected CDN libraries
grep -c 'chart.js' bundle/apps/runtime.html && \
grep -c 'leaflet' bundle/apps/runtime.html && \
grep -c 'react@19' bundle/apps/runtime.html && \
grep -c 'babel/standalone' bundle/apps/runtime.html && \
echo "All libraries referenced in runtime.html"
```
Expected: 4 counts (one per library family) and "All libraries referenced".
**Step 4: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add bundle/ && git commit -m "feat: complete bundle with visual artifact runtime and context"
```
---
### Task 8: Browser panel component (noVNC iframe)
**Files:**
- Create: `frontend/src/components/BrowserPanel.tsx`
**Step 1: Create the browser panel**
Create `frontend/src/components/BrowserPanel.tsx`:
```tsx
/**
* Browser panel: embeds a noVNC iframe showing the live Playwright browser.
*
* In production (behind cloudflared): points to vnc.ampbox.io
* In local dev (Docker): points to localhost:6080
* The URL is determined by the current hostname.
*/
interface BrowserPanelProps {
sessionId: string | null;
}
function getVncUrl(): string {
const hostname = window.location.hostname;
// Production: behind cloudflared
if (hostname === "research.ampbox.io") {
return "https://vnc.ampbox.io/vnc.html?autoconnect=true&resize=remote&quality=6&compression=2";
}
// Local Docker: noVNC on port 6080
return `http://${hostname}:6080/vnc.html?autoconnect=true&resize=remote&quality=6&compression=2`;
}
export function BrowserPanel({ sessionId }: BrowserPanelProps) {
if (!sessionId) {
return (
<div style={{ padding: "2rem", color: "#6a6a8a", textAlign: "center" }}>
Select a session to view the browser
</div>
);
}
return (
<iframe
src={getVncUrl()}
title="Browser - noVNC"
style={{
width: "100%",
height: "100%",
border: "none",
backgroundColor: "#0a0a1a",
}}
allow="clipboard-read; clipboard-write"
/>
);
}
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add frontend/src/components/BrowserPanel.tsx && git commit -m "feat: browser panel with noVNC iframe"
```
---
### Task 9: Artifacts panel component
**Files:**
- Create: `frontend/src/components/ArtifactsPanel.tsx`
**Step 1: Create the artifacts panel**
Create `frontend/src/components/ArtifactsPanel.tsx`:
```tsx
import { useEffect } from "react";
import ReactMarkdown from "react-markdown";
import { IconFileText, IconRefresh } from "@tabler/icons-react";
import { useArtifacts } from "../hooks/useArtifacts";
interface ArtifactsPanelProps {
sessionId: string | null;
}
export function ArtifactsPanel({ sessionId }: ArtifactsPanelProps) {
const { artifacts, activeArtifact, refresh, loadArtifact } = useArtifacts(sessionId);
// Refresh artifact list when session changes
useEffect(() => {
if (sessionId) refresh();
}, [sessionId, refresh]);
if (!sessionId) {
return (
<div style={{ padding: "2rem", color: "#6a6a8a", textAlign: "center" }}>
Select a session to view artifacts
</div>
);
}
return (
<div style={{ display: "flex", flexDirection: "column", height: "100%" }}>
{/* Artifact list header */}
<div
style={{
padding: "0.5rem 0.75rem",
borderBottom: "1px solid #2a2a4a",
display: "flex",
justifyContent: "space-between",
alignItems: "center",
}}
>
<span style={{ fontSize: "0.8rem", color: "#8888aa" }}>
{artifacts.length} artifact{artifacts.length !== 1 ? "s" : ""}
</span>
<button
onClick={refresh}
style={{
background: "none",
border: "none",
color: "#7aa2f7",
cursor: "pointer",
padding: "2px",
}}
title="Refresh"
>
<IconRefresh size={16} />
</button>
</div>
{/* Artifact tabs */}
{artifacts.length > 0 && (
<div
style={{
display: "flex",
gap: "2px",
padding: "0.3rem 0.5rem",
overflowX: "auto",
borderBottom: "1px solid #2a2a4a",
}}
>
{artifacts.map((a) => (
<button
key={a.name}
onClick={() => loadArtifact(a.name)}
style={{
background:
activeArtifact?.name === a.name ? "#2a3a5a" : "transparent",
border: "none",
color: activeArtifact?.name === a.name ? "#ffffff" : "#8888aa",
padding: "0.3rem 0.6rem",
borderRadius: "4px",
cursor: "pointer",
fontSize: "0.75rem",
whiteSpace: "nowrap",
display: "flex",
alignItems: "center",
gap: "4px",
}}
>
<IconFileText size={14} />
{a.name}
</button>
))}
</div>
)}
{/* Artifact content */}
<div
style={{
flex: 1,
overflowY: "auto",
padding: "1rem",
}}
>
{activeArtifact ? (
<div className="chat-message assistant">
<ReactMarkdown>{activeArtifact.content}</ReactMarkdown>
</div>
) : artifacts.length > 0 ? (
<div style={{ color: "#6a6a8a", textAlign: "center", paddingTop: "2rem" }}>
Click an artifact tab to view it
</div>
) : (
<div style={{ color: "#6a6a8a", textAlign: "center", paddingTop: "2rem" }}>
No artifacts yet. The researcher will create them during research.
</div>
)}
</div>
</div>
);
}
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add frontend/src/components/ArtifactsPanel.tsx && git commit -m "feat: artifacts panel with tabbed markdown rendering"
```
---
### Task 10: A2UI renderer component
Parses `a2ui` JSON code blocks from assistant messages and renders them as interactive Tabler components.
**Files:**
- Create: `frontend/src/components/A2UIRenderer.tsx`
**Step 1: Create the A2UI renderer**
Create `frontend/src/components/A2UIRenderer.tsx`:
```tsx
import type { A2UIBlock } from "../lib/types";
import { IconFilter, IconTable, IconCards, IconForms } from "@tabler/icons-react";
interface A2UIRendererProps {
block: A2UIBlock;
onAction?: (action: string, data: Record<string, unknown>) => void;
}
export function A2UIRenderer({ block, onAction }: A2UIRendererProps) {
switch (block.type) {
case "filters":
return (
<div className="tool-call-card" style={{ display: "flex", gap: "0.4rem", flexWrap: "wrap", alignItems: "center" }}>
<IconFilter size={16} style={{ color: "#7aa2f7", marginRight: 4 }} />
{block.options.map((opt, i) => (
<button
key={i}
onClick={() => onAction?.("filter_toggle", { label: opt.label, active: !opt.active })}
style={{
padding: "0.3rem 0.6rem",
borderRadius: "12px",
border: "1px solid",
borderColor: opt.active ? "#5a7aba" : "#3a3a5a",
backgroundColor: opt.active ? "#2a3a6a" : "transparent",
color: opt.active ? "#ffffff" : "#8888aa",
cursor: "pointer",
fontSize: "0.8rem",
}}
>
{opt.label}
</button>
))}
</div>
);
case "table":
return (
<div className="tool-call-card" style={{ overflowX: "auto" }}>
<div style={{ marginBottom: "0.4rem", display: "flex", alignItems: "center", gap: 4 }}>
<IconTable size={16} style={{ color: "#7aa2f7" }} />
<span style={{ fontSize: "0.8rem", color: "#8888aa" }}>Data Table</span>
</div>
<table style={{ width: "100%", borderCollapse: "collapse", fontSize: "0.8rem" }}>
<thead>
<tr>
{block.headers.map((h, i) => (
<th
key={i}
style={{
textAlign: "left",
padding: "0.3rem 0.5rem",
borderBottom: "1px solid #2a2a4a",
color: "#aaaacc",
fontWeight: 600,
}}
>
{h}
</th>
))}
</tr>
</thead>
<tbody>
{block.rows.map((row, ri) => (
<tr key={ri}>
{row.map((cell, ci) => (
<td
key={ci}
style={{
padding: "0.3rem 0.5rem",
borderBottom: "1px solid #1a1a2e",
color: "#c0c0d0",
}}
>
{cell}
</td>
))}
</tr>
))}
</tbody>
</table>
</div>
);
case "cards":
return (
<div style={{ display: "grid", gridTemplateColumns: "repeat(auto-fill, minmax(200px, 1fr))", gap: "0.5rem", margin: "0.5rem 0" }}>
{block.items.map((item, i) => (
<div
key={i}
className="tool-call-card"
style={{ cursor: item.url ? "pointer" : "default" }}
onClick={() => item.url && window.open(item.url, "_blank")}
>
<div style={{ fontWeight: 600, color: "#e0e0e0", marginBottom: 4 }}>{item.title}</div>
{item.price && (
<div style={{ color: "#7af7a2", fontWeight: 600, fontSize: "0.9rem" }}>{item.price}</div>
)}
{item.description && (
<div style={{ color: "#8888aa", fontSize: "0.75rem", marginTop: 4 }}>{item.description}</div>
)}
</div>
))}
</div>
);
case "form":
return (
<div className="tool-call-card">
<div style={{ marginBottom: "0.5rem", display: "flex", alignItems: "center", gap: 4 }}>
<IconForms size={16} style={{ color: "#7aa2f7" }} />
<span style={{ fontSize: "0.8rem", color: "#8888aa" }}>Input Form</span>
</div>
<form
onSubmit={(e) => {
e.preventDefault();
const formData = new FormData(e.currentTarget);
const data: Record<string, unknown> = {};
formData.forEach((v, k) => (data[k] = v));
onAction?.("form_submit", data);
}}
style={{ display: "flex", flexDirection: "column", gap: "0.4rem" }}
>
{block.fields.map((field) => (
<div key={field.name}>
<label style={{ fontSize: "0.75rem", color: "#8888aa", display: "block", marginBottom: 2 }}>
{field.label || field.name}
</label>
{field.type === "select" ? (
<select
name={field.name}
defaultValue={String(field.value ?? "")}
style={{
width: "100%",
padding: "0.3rem",
backgroundColor: "#1e2a4a",
border: "1px solid #3a3a5a",
borderRadius: "4px",
color: "#e0e0e0",
fontSize: "0.8rem",
}}
>
{field.options?.map((opt) => (
<option key={opt} value={opt}>{opt}</option>
))}
</select>
) : (
<input
name={field.name}
type={field.type}
defaultValue={String(field.value ?? "")}
style={{
width: "100%",
padding: "0.3rem",
backgroundColor: "#1e2a4a",
border: "1px solid #3a3a5a",
borderRadius: "4px",
color: "#e0e0e0",
fontSize: "0.8rem",
boxSizing: "border-box",
}}
/>
)}
</div>
))}
<button
type="submit"
style={{
padding: "0.4rem 0.8rem",
backgroundColor: "#3a5aba",
color: "white",
border: "none",
borderRadius: "4px",
cursor: "pointer",
fontSize: "0.8rem",
marginTop: "0.3rem",
alignSelf: "flex-start",
}}
>
{block.submitLabel || "Submit"}
</button>
</form>
</div>
);
default:
return null;
}
}
/**
* Parse A2UI blocks from message content.
* Looks for ```a2ui ... ``` fenced code blocks and extracts the JSON.
*/
export function parseA2UIBlocks(content: string): A2UIBlock[] {
const blocks: A2UIBlock[] = [];
const regex = /```a2ui\s*\n([\s\S]*?)```/g;
let match;
while ((match = regex.exec(content)) !== null) {
try {
const parsed = JSON.parse(match[1].trim());
if (parsed && parsed.type) {
blocks.push(parsed as A2UIBlock);
}
} catch {
// Skip unparseable blocks
}
}
return blocks;
}
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add frontend/src/components/A2UIRenderer.tsx && git commit -m "feat: A2UI renderer for filters, tables, cards, forms"
```
---
### Task 11: Wire up RightPanel and integrate A2UI + visual blocks into ChatPanel
**Files:**
- Modify: `frontend/src/components/RightPanel.tsx`
- Modify: `frontend/src/components/ChatPanel.tsx`
**Step 1: Update RightPanel with real Browser and Artifacts components**
Replace `frontend/src/components/RightPanel.tsx` with:
```tsx
import { useState } from "react";
import { BrowserPanel } from "./BrowserPanel";
import { ArtifactsPanel } from "./ArtifactsPanel";
interface RightPanelProps {
sessionId: string | null;
}
export function RightPanel({ sessionId }: RightPanelProps) {
const [activeTab, setActiveTab] = useState<"browser" | "artifacts">("browser");
return (
<div className="right-panel">
<div className="right-panel-tabs">
<button
className={activeTab === "browser" ? "active" : ""}
onClick={() => setActiveTab("browser")}
>
Browser
</button>
<button
className={activeTab === "artifacts" ? "active" : ""}
onClick={() => setActiveTab("artifacts")}
>
Artifacts
</button>
</div>
<div className="right-panel-content">
{activeTab === "browser" ? (
<BrowserPanel sessionId={sessionId} />
) : (
<ArtifactsPanel sessionId={sessionId} />
)}
</div>
</div>
);
}
```
**Step 2: Update ChatPanel to render A2UI blocks and visual blocks**
In `frontend/src/components/ChatPanel.tsx`, add the A2UI import and rendering. Replace the message rendering section inside the `messages.map()` where assistant messages are rendered. Specifically, after the `<ReactMarkdown>` call, add A2UI parsing:
Add this import at the top:
```tsx
import { A2UIRenderer, parseA2UIBlocks } from "./A2UIRenderer";
```
Then in the assistant message rendering block (inside `messages.map`), replace:
```tsx
{msg.role === "assistant" ? (
<>
<ReactMarkdown>{msg.content}</ReactMarkdown>
{msg.toolCalls?.map((tc) => (
<ToolCallCard key={tc.id} toolCall={tc} />
))}
</>
```
With:
```tsx
{msg.role === "assistant" ? (
<>
<ReactMarkdown>{msg.content}</ReactMarkdown>
{parseA2UIBlocks(msg.content).map((block, i) => (
<A2UIRenderer
key={i}
block={block}
onAction={(action, data) => {
// Send A2UI interaction back as user message
onSend(`[${action}] ${JSON.stringify(data)}`);
}}
/>
))}
{msg.toolCalls?.map((tc) => (
<ToolCallCard key={tc.id} toolCall={tc} />
))}
</>
```
**Step 3: Verify the build compiles**
Run:
```bash
cd /home/ken/workspace/research-workbench/frontend
npm run build 2>&1 | tail -10
```
Expected: Build succeeds.
**Step 4: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add frontend/src/components/ && git commit -m "feat: wire RightPanel with Browser + Artifacts, add A2UI rendering to chat"
```
---
### Task 12: Docker Compose for local development
**Files:**
- Create: `docker-compose.yml`
**Step 1: Create docker-compose.yml**
Create `docker-compose.yml`:
```yaml
# Local development: docker compose up
# Builds and runs the all-in-one container.
#
# Before running:
# 1. Build the frontend: cd frontend && npm run build
# 2. Set your API key: export ANTHROPIC_API_KEY=sk-...
# 3. Generate a password hash: python -c "import bcrypt; print(bcrypt.hashpw(b'YOUR_PASSWORD', bcrypt.gensalt()).decode())"
services:
workbench:
build: .
ports:
- "8080:8080" # Web UI
- "6080:6080" # noVNC browser view
environment:
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- AUTH_USER=${AUTH_USER:-admin@localhost}
- AUTH_PASS_HASH=${AUTH_PASS_HASH}
- DISPLAY=:99
volumes:
- ./artifacts:/app/artifacts # Persist artifacts across restarts
restart: unless-stopped
```
**Step 2: Commit**
Run:
```bash
cd /home/ken/workspace/research-workbench
git add docker-compose.yml && git commit -m "feat: docker-compose.yml for local development"
```
---
## Cross-Phase Dependencies
The visual artifact system introduced in Tasks 5-6 requires coordinating changes across all three phases. These are notes for the implementer:
### Phase 1 Backend (already handled)
The backend already serves `GET /api/apps/{app_name}` which reads HTML files from `bundle/apps/`. Since we replaced 4 template files with a single `runtime.html`, the endpoint continues to work — the frontend will request `/api/apps/runtime` and get `runtime.html`.
The AG-UI adapter already passes through `_meta.ui` and `structuredContent` from tool results. No changes needed.
### Phase 2 Frontend (needs updating when implementing Phase 3)
The `MCPAppRenderer` component (Phase 2, Task 9) and `ChatPanel` (Phase 2, Tasks 7 + 10) need these updates to support `visual` code blocks:
1. **Detect `visual` code blocks in assistant messages**: Add a `parseVisualBlocks(content)` function (similar to `parseA2UIBlocks`) that extracts `` ```visual `` fenced code blocks and parses the JSON (`{ title, code, data, height }`).
2. **Create a `VisualBlock` wrapper component** that:
- Displays the visual's `title` in a header bar
- Loads `runtime.html` from `/api/apps/runtime` via the existing `MCPAppRenderer`
- After the iframe signals `ui/initialize`, sends the `render` method via postMessage with the visual's `code` and `data`
- Supports a "pop out" button to open the visual in a full browser tab
- Supports a "view source" toggle to show the generated code
3. **Wire `VisualBlock` into `ChatPanel`**: After A2UI blocks, render any detected visual blocks from the assistant message content.
4. **Update `MCPAppRenderer`**: Add support for the `render` postMessage method (in addition to existing `tool/input`). When the iframe sends `ui/initialize` with `runtime: "visual-artifact"`, send `{ method: "render", params: { code, data } }` instead of `{ method: "tool/input", params: data }`.
These updates should be implemented during Phase 3 Task 11 (which already modifies `ChatPanel.tsx`).
---
## Phase 3 Complete
At this point you have:
- **Bundle**: Thin root including foundation, with researcher agent, A2UI format spec, artifact conventions, and visual artifact generation context
- **Visual artifact runtime**: Single `runtime.html` that executes arbitrary AI-generated React/HTML/JS code in a sandboxed iframe with pre-loaded Chart.js, Leaflet, React 19, Babel, and Tabler CSS
- **Visual artifacts context**: Agent instructions with 5 complete example patterns (comparison table, bar chart, location map, React dashboard, decision matrix with interactive weight sliders)
- **Browser panel**: noVNC iframe with auto-detection of production vs local URLs
- **Artifacts panel**: Tabbed markdown rendering of per-session artifacts
- **A2UI renderer**: Interactive filters, tables, cards, and forms from agent-generated JSON
- **Docker Compose**: One-command local deployment
**The AI-generated visuals flow:**
```
Agent writes a ```visual code block in its message
-> Frontend detects the visual block, parses JSON { title, code, data, height }
-> VisualBlock component loads runtime.html from /api/apps/runtime
-> After iframe ready (ui/initialize), sends { method: "render", params: { code, data } }
-> runtime.html receives code, auto-detects format (JSX/HTML/JS)
-> Compiles JSX via Babel if needed, executes code with data in scope
-> Renders interactive chart/table/map/dashboard in the iframe
-> User clicks elements -> mcp.sendMessage() injects follow-up into chat
```
**Full deployment flow:**
```bash
# 1. Build frontend
cd frontend && npm run build && cd ..
# 2. Build and run Docker container
export ANTHROPIC_API_KEY=sk-...
export AUTH_USER=ken@example.com
export AUTH_PASS_HASH=$(python3 -c "import bcrypt; print(bcrypt.hashpw(b'mypassword', bcrypt.gensalt()).decode())")
docker compose up --build
# 3. Open http://localhost:8080
# 4. Log in with your credentials
# 5. Create a session and start researching!
```
**Production deployment (vela0 + cloudflared):**
The container exposes ports 8080 and 6080. The existing `/mnt/services/research.yml` cloudflared config routes:
- `research.ampbox.io` -> `http://10.66.204.209:8080`
- `vnc.ampbox.io` -> `http://10.66.204.209:6080`