4656af010e
Phase 1: Infrastructure (8 tasks) - Dockerfile, amplifierd, FastAPI, auth, AG-UI adapter Phase 2: Frontend (10 tasks) - React/TS/Vite, Tabler, CopilotKit headless, SSE streaming Phase 3: Bundle + Browser + Artifacts (12 tasks) - Amplifier bundle, researcher agent, AI-generated visual artifacts (Claude builds visuals style), noVNC browser panel
1586 lines
57 KiB
Markdown
1586 lines
57 KiB
Markdown
# Research Workbench — Phase 3: Bundle + Browser + Artifacts
|
|
|
|
> **Execution:** Use the subagent-driven-development workflow to implement this plan.
|
|
|
|
**Goal:** Create the Amplifier bundle (researcher agent), wire up the browser panel (noVNC iframe), implement the artifacts panel with markdown rendering, add A2UI block rendering for inline AI-generated UI controls, and build the visual artifact runtime for AI-generated interactive inline visualizations (Claude "builds visuals" style).
|
|
|
|
**Architecture:** The bundle follows the thin-root pattern: `bundle.md` includes foundation (which brings tool-bash, tool-filesystem, tool-web-fetch) and declares the researcher agent. The agent's body contains playwright-cli command reference, A2UI output format spec, artifact conventions, and visual artifact generation instructions. A single universal runtime (`bundle/apps/runtime.html`) executes arbitrary AI-generated code (React/JSX, Chart.js, Leaflet, plain HTML) in a sandboxed iframe. The AI generates visualization code on the fly during conversation — not from fixed templates. The browser panel embeds noVNC via iframe. The artifacts panel lists and renders markdown files written by the agent.
|
|
|
|
**Tech Stack:** Amplifier bundle (markdown + YAML), React components (Tabler), react-markdown, noVNC iframe, Chart.js, Leaflet.js, React 19/Babel standalone (in-browser JSX), MCP Apps (SEP-1865)
|
|
|
|
**Depends on:** Phase 1 (backend) and Phase 2 (frontend shell) complete
|
|
|
|
---
|
|
|
|
### Task 1: Bundle root (bundle.md)
|
|
|
|
The thin root bundle. It includes foundation (which provides tool-bash, tool-filesystem, tool-web-fetch) and declares the researcher agent. Do NOT redeclare foundation tools — that's the canonical anti-pattern.
|
|
|
|
**Files:**
|
|
- Create: `bundle/bundle.md`
|
|
|
|
**Step 1: Create bundle.md**
|
|
|
|
Create `bundle/bundle.md`:
|
|
|
|
```markdown
|
|
---
|
|
bundle:
|
|
name: research-workbench
|
|
version: 0.1.0
|
|
description: General-purpose AI research tool with browser control and artifact generation
|
|
|
|
includes:
|
|
- bundle: git+https://github.com/microsoft/amplifier-foundation@main
|
|
- bundle: research-workbench:behaviors/research-workbench
|
|
---
|
|
|
|
# Research Workbench
|
|
|
|
A research-first Amplifier bundle. Provides a researcher agent with browser
|
|
control (via playwright-cli), artifact generation, and A2UI inline controls.
|
|
|
|
## What's Included
|
|
|
|
**From foundation:** tool-bash, tool-filesystem, tool-web-fetch, and all standard hooks.
|
|
|
|
**From this bundle:**
|
|
- `researcher` agent — browser-equipped research assistant
|
|
- Research workbench behavior — wires the researcher agent with context
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add bundle/bundle.md && git commit -m "feat: thin root bundle including foundation"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: Research workbench behavior
|
|
|
|
**Files:**
|
|
- Create: `bundle/behaviors/research-workbench.yaml`
|
|
|
|
**Step 1: Create the behavior file**
|
|
|
|
Create `bundle/behaviors/research-workbench.yaml`:
|
|
|
|
```yaml
|
|
# research-workbench behavior
|
|
# Wires the researcher agent and research context into sessions.
|
|
|
|
behavior:
|
|
name: research-workbench
|
|
description: Browser-equipped research assistant with artifact generation
|
|
|
|
agents:
|
|
include:
|
|
- researcher
|
|
|
|
context:
|
|
include:
|
|
- research-workbench:context/researcher-instructions
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add bundle/behaviors/research-workbench.yaml && git commit -m "feat: research-workbench behavior wiring researcher agent"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Researcher agent definition
|
|
|
|
The researcher agent is the core of the system. It knows how to use playwright-cli for browser control, generate A2UI blocks for interactive controls, and create artifacts.
|
|
|
|
**Files:**
|
|
- Create: `bundle/agents/researcher.md`
|
|
|
|
**Step 1: Create the researcher agent**
|
|
|
|
Create `bundle/agents/researcher.md`:
|
|
|
|
```markdown
|
|
---
|
|
meta:
|
|
name: researcher
|
|
description: |
|
|
WHY: Users need a hands-free research assistant that can browse the web,
|
|
extract data, compare options, and generate polished deliverables.
|
|
|
|
WHEN: Delegated to for any research task — product comparisons, market
|
|
research, technical investigations, travel planning, buying guides.
|
|
|
|
WHAT: Conducts multi-step web research using a real browser (playwright-cli),
|
|
generates interactive inline UI controls (A2UI), and produces artifacts
|
|
(guides, reports, comparison tables) as markdown files.
|
|
|
|
HOW: Opens browser tabs via playwright-cli, takes compact snapshots,
|
|
clicks elements, fills forms. Generates A2UI JSON blocks inline when the
|
|
user would benefit from filters, forms, or data tables. Writes artifacts
|
|
to the artifacts directory for persistent deliverables.
|
|
|
|
Examples of delegation:
|
|
- "Research hybrid cars under $10K near Woodinville WA"
|
|
- "Compare the top 5 project management tools for small teams"
|
|
- "Find flights from Seattle to Tokyo in October under $800"
|
|
- "Investigate the best mechanical keyboards for programming"
|
|
model_role:
|
|
- research
|
|
- general
|
|
---
|
|
|
|
# Researcher Agent
|
|
|
|
You are a research assistant with a live web browser. You help users
|
|
investigate topics, compare options, and produce polished deliverables.
|
|
|
|
## Browser Control (playwright-cli)
|
|
|
|
You control a Chromium browser via `playwright-cli`. The browser is visible
|
|
to the user in real-time through a VNC panel.
|
|
|
|
### Commands
|
|
|
|
All commands use `playwright-cli -s=research` (the `-s` flag keeps a persistent
|
|
browser session across commands).
|
|
|
|
| Action | Command |
|
|
|--------|---------|
|
|
| Open a URL | `playwright-cli -s=research open https://example.com` |
|
|
| Take a snapshot | `playwright-cli -s=research snapshot` |
|
|
| Compact snapshot | `playwright-cli -s=research snapshot -ic` |
|
|
| Click an element | `playwright-cli -s=research click @e5` |
|
|
| Type text | `playwright-cli -s=research type @e3 "search query"` |
|
|
| Press a key | `playwright-cli -s=research press Enter` |
|
|
| Scroll down | `playwright-cli -s=research scroll down` |
|
|
| Go back | `playwright-cli -s=research back` |
|
|
| New tab | `playwright-cli -s=research open https://other-site.com` |
|
|
| List tabs | `playwright-cli -s=research tabs` |
|
|
| Switch tab | `playwright-cli -s=research tab 2` |
|
|
|
|
### Snapshot Refs
|
|
|
|
Snapshots return an accessibility tree with element refs like `@e1`, `@e2`, etc.
|
|
Use these refs for click/type actions. Always take a snapshot after navigation
|
|
to see the page state.
|
|
|
|
### Best Practices
|
|
|
|
1. Always take a compact snapshot (`-ic`) after opening a URL or clicking
|
|
2. Use refs from the most recent snapshot — they change after navigation
|
|
3. Open new sites in new tabs (both URLs will be visible in the VNC panel)
|
|
4. The user can see and interact with the browser via VNC at any time
|
|
|
|
## A2UI (AI-to-UI) Blocks
|
|
|
|
You can generate interactive UI controls inline in your messages. Wrap them
|
|
in a JSON code block with the `a2ui` language tag. The frontend renders them
|
|
as Tabler components.
|
|
|
|
@mention bundle/context/a2ui-format.md
|
|
|
|
## Artifacts
|
|
|
|
When you produce a substantial deliverable (guide, report, comparison table),
|
|
save it as a markdown file in the artifacts directory.
|
|
|
|
@mention bundle/context/artifact-conventions.md
|
|
|
|
## Inline Visuals (Claude-style)
|
|
|
|
You can generate interactive visualizations inline in your messages. When your
|
|
research would benefit from a visual (comparison chart, data table, map, dashboard,
|
|
decision matrix), write the code directly as a `visual` code block. The frontend
|
|
renders it live in a sandboxed iframe.
|
|
|
|
@mention bundle/context/visual-artifacts.md
|
|
|
|
Be proactive: if you're comparing 5 products, generate a comparison table.
|
|
If you're showing locations, generate a map. If you're showing trends, generate
|
|
a chart. Don't wait to be asked.
|
|
|
|
## Research Workflow
|
|
|
|
1. **Understand the request** — clarify scope, constraints, preferences
|
|
2. **Plan the research** — identify sites to check, data to gather
|
|
3. **Browse and extract** — open sites, take snapshots, extract data
|
|
4. **Present findings** — use A2UI blocks for simple controls, `visual` blocks for rich interactive visualizations
|
|
5. **Generate artifacts** — write polished guides/reports as markdown files
|
|
6. **Iterate** — refine based on user feedback, apply filters, dig deeper
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add bundle/agents/researcher.md && git commit -m "feat: researcher agent with playwright-cli, A2UI, inline visuals, artifact conventions"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Context files (A2UI format, artifact conventions, instructions)
|
|
|
|
**Files:**
|
|
- Create: `bundle/context/a2ui-format.md`
|
|
- Create: `bundle/context/artifact-conventions.md`
|
|
- Create: `bundle/context/researcher-instructions.md`
|
|
|
|
**Step 1: Create a2ui-format.md**
|
|
|
|
Create `bundle/context/a2ui-format.md`:
|
|
|
|
````markdown
|
|
# A2UI Block Format
|
|
|
|
Generate interactive UI by emitting a fenced code block with language `a2ui`
|
|
containing a JSON object. The frontend parses these and renders Tabler
|
|
components.
|
|
|
|
## Supported Block Types
|
|
|
|
### Filters
|
|
|
|
Toggle buttons for filtering results. When the user clicks a filter, the
|
|
frontend sends a message with the updated filter state.
|
|
|
|
```a2ui
|
|
{
|
|
"type": "filters",
|
|
"options": [
|
|
{"label": "Toyota", "active": true},
|
|
{"label": "Honda", "active": true},
|
|
{"label": "Hyundai", "active": false},
|
|
{"label": "Under $5K", "active": false}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Table
|
|
|
|
Data table with headers and rows. Good for comparisons.
|
|
|
|
```a2ui
|
|
{
|
|
"type": "table",
|
|
"headers": ["Model", "Year", "Price", "Miles", "Source"],
|
|
"rows": [
|
|
["Prius", "2014", "$6,500", "89K", "Craigslist"],
|
|
["Camry Hybrid", "2012", "$7,200", "102K", "CarGurus"]
|
|
]
|
|
}
|
|
```
|
|
|
|
### Cards
|
|
|
|
Card grid for items with images (listings, products, places).
|
|
|
|
```a2ui
|
|
{
|
|
"type": "cards",
|
|
"items": [
|
|
{
|
|
"title": "2014 Toyota Prius",
|
|
"description": "Clean title, one owner, 89K miles",
|
|
"price": "$6,500",
|
|
"url": "https://example.com/listing/123"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Form
|
|
|
|
Input form for structured user preferences.
|
|
|
|
```a2ui
|
|
{
|
|
"type": "form",
|
|
"fields": [
|
|
{"name": "max_price", "type": "number", "label": "Max Price", "value": 10000},
|
|
{"name": "radius", "type": "number", "label": "Search Radius (mi)", "value": 50},
|
|
{"name": "fuel_type", "type": "select", "label": "Fuel", "options": ["hybrid", "electric", "any"]}
|
|
],
|
|
"submitLabel": "Update Search"
|
|
}
|
|
```
|
|
|
|
## Rules
|
|
|
|
- Always use the `a2ui` language tag on the code fence
|
|
- The JSON must be a single object with a `type` field
|
|
- Generate A2UI blocks when the user would benefit from interactive controls
|
|
- Don't overuse — plain text is fine for simple responses
|
|
````
|
|
|
|
**Step 2: Create artifact-conventions.md**
|
|
|
|
Create `bundle/context/artifact-conventions.md`:
|
|
|
|
```markdown
|
|
# Artifact Conventions
|
|
|
|
Artifacts are polished deliverables saved as markdown files. They persist
|
|
across the session and are viewable in the Artifacts panel.
|
|
|
|
## Writing Artifacts
|
|
|
|
Use tool-filesystem to write files to the artifacts directory:
|
|
|
|
```
|
|
write_file("artifacts/<session_id>/<name>.md", content)
|
|
```
|
|
|
|
The `<session_id>` matches the current session. Ask the system for the
|
|
session ID if you don't know it, or use the working directory path.
|
|
|
|
## File Naming
|
|
|
|
- Use kebab-case: `buying-guide.md`, `comparison-table.md`
|
|
- Be descriptive: `hybrid-cars-under-10k-guide.md`
|
|
- Always use `.md` extension
|
|
|
|
## Content Format
|
|
|
|
- Start with an H1 title
|
|
- Include a date and scope summary at the top
|
|
- Use markdown tables for comparisons
|
|
- Use bullet points for recommendations
|
|
- Include source URLs where applicable
|
|
- End with a "Next Steps" or "Summary" section
|
|
|
|
## When to Create Artifacts
|
|
|
|
Create an artifact when:
|
|
- The user asks for a guide, report, or summary
|
|
- You've gathered enough data for a comparison table
|
|
- The research produces actionable recommendations
|
|
- The user would benefit from a printable/shareable document
|
|
|
|
Don't create artifacts for:
|
|
- Simple answers or clarifications
|
|
- Intermediate research steps
|
|
- Status updates
|
|
```
|
|
|
|
**Step 3: Create researcher-instructions.md (thin awareness pointer)**
|
|
|
|
Create `bundle/context/researcher-instructions.md`:
|
|
|
|
```markdown
|
|
# Research Workbench Context
|
|
|
|
This session runs inside the Research Workbench. You have access to:
|
|
|
|
- **Browser**: A live Chromium browser controlled via `playwright-cli -s=research`.
|
|
The user can see the browser in the VNC panel and take over at any time.
|
|
- **Artifacts**: Write research deliverables to `artifacts/<session_id>/`.
|
|
They appear in the Artifacts panel.
|
|
- **A2UI**: Generate inline interactive controls using `a2ui` code blocks.
|
|
- **Inline Visuals**: Generate interactive charts, tables, maps, and dashboards
|
|
using `visual` code blocks. See the researcher agent for full details.
|
|
|
|
See the researcher agent definition for full command reference.
|
|
```
|
|
|
|
**Step 4: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add bundle/context/ && git commit -m "feat: context files for A2UI format, artifact conventions, researcher instructions"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: Visual artifact runtime (AI-generated inline visuals)
|
|
|
|
A single universal runtime that executes AI-generated visualization code in a sandboxed iframe. Instead of fixed templates, the AI generates arbitrary React/HTML/JS code during conversation (Claude "builds visuals" style), and this runtime renders it live. Pre-loads common visualization libraries so the AI can use them immediately.
|
|
|
|
**Files:**
|
|
- Create: `bundle/apps/runtime.html`
|
|
|
|
**Step 1: Create runtime.html**
|
|
|
|
Create `bundle/apps/runtime.html`:
|
|
|
|
```html
|
|
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
|
|
<!-- Pre-loaded libraries available to all AI-generated visuals -->
|
|
<link href="https://cdn.jsdelivr.net/npm/@tabler/core@1.0.0-beta20/dist/css/tabler.min.css" rel="stylesheet">
|
|
<link href="https://cdn.jsdelivr.net/npm/leaflet@1.9/dist/leaflet.css" rel="stylesheet">
|
|
|
|
<script src="https://cdn.jsdelivr.net/npm/chart.js@4"></script>
|
|
<script src="https://cdn.jsdelivr.net/npm/leaflet@1.9/dist/leaflet.js"></script>
|
|
<script src="https://cdn.jsdelivr.net/npm/react@19/umd/react.production.min.js"></script>
|
|
<script src="https://cdn.jsdelivr.net/npm/react-dom@19/umd/react-dom.production.min.js"></script>
|
|
<script src="https://cdn.jsdelivr.net/npm/@babel/standalone@7/babel.min.js"></script>
|
|
|
|
<style>
|
|
body {
|
|
background: transparent;
|
|
margin: 0;
|
|
padding: 16px;
|
|
color: #e0e0e0;
|
|
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
|
}
|
|
.theme-dark { color-scheme: dark; }
|
|
.visual-error {
|
|
color: #f7607a;
|
|
background: #2a1a1e;
|
|
border: 1px solid #4a2a3a;
|
|
border-radius: 8px;
|
|
padding: 1rem;
|
|
font-family: monospace;
|
|
font-size: 0.8rem;
|
|
white-space: pre-wrap;
|
|
}
|
|
</style>
|
|
</head>
|
|
<body class="theme-dark">
|
|
<div id="root"></div>
|
|
|
|
<script>
|
|
// --- MCP Apps JSON-RPC 2.0 bridge (pre-wired) ---
|
|
const mcpBridge = {
|
|
_pending: new Map(),
|
|
|
|
sendToHost(msg) {
|
|
window.parent.postMessage(msg, "*");
|
|
},
|
|
|
|
callTool(name, args) {
|
|
const id = crypto.randomUUID();
|
|
this.sendToHost({
|
|
jsonrpc: "2.0", id,
|
|
method: "tools/call",
|
|
params: { name, arguments: args },
|
|
});
|
|
return new Promise((resolve) => this._pending.set(id, resolve));
|
|
},
|
|
|
|
sendMessage(content) {
|
|
this.sendToHost({
|
|
jsonrpc: "2.0",
|
|
method: "ui/sendMessage",
|
|
params: { role: "user", content },
|
|
});
|
|
},
|
|
|
|
resize(height) {
|
|
this.sendToHost({
|
|
jsonrpc: "2.0",
|
|
method: "ui/resize",
|
|
params: { height: Math.min(height, 800) },
|
|
});
|
|
},
|
|
};
|
|
|
|
// --- Visual execution engine ---
|
|
|
|
function executeVisual(code, data) {
|
|
const root = document.getElementById("root");
|
|
|
|
try {
|
|
// Make data and bridge available to the generated code
|
|
window.__VISUAL_DATA__ = data || {};
|
|
window.__MCP__ = mcpBridge;
|
|
|
|
// Detect if the code contains JSX (look for < followed by a capital letter
|
|
// or common HTML tags used in React-like syntax)
|
|
const hasJSX = /<[A-Z]/.test(code) || /React\.createElement/.test(code);
|
|
const hasBabelType = /type\s*=\s*["']text\/babel["']/.test(code);
|
|
|
|
if (hasJSX || hasBabelType) {
|
|
// --- React/JSX path: compile with Babel, render to #root ---
|
|
const compiled = Babel.transform(code, {
|
|
presets: ["react"],
|
|
filename: "visual.jsx",
|
|
}).code;
|
|
|
|
// Wrap in a function that has access to React, ReactDOM, data, and bridge
|
|
const fn = new Function(
|
|
"React", "ReactDOM", "data", "mcp", "root",
|
|
compiled + "\n\n" +
|
|
"// Auto-render: if code defined a default export or App component\n" +
|
|
"if (typeof App !== 'undefined') {\n" +
|
|
" ReactDOM.createRoot(root).render(React.createElement(App, { data, mcp }));\n" +
|
|
"}\n"
|
|
);
|
|
fn(React, ReactDOM, data, mcpBridge, root);
|
|
} else if (/<[a-z][\s\S]*>/i.test(code)) {
|
|
// --- HTML path: inject directly ---
|
|
root.innerHTML = code;
|
|
|
|
// Execute any <script> tags in the injected HTML
|
|
root.querySelectorAll("script").forEach((oldScript) => {
|
|
const newScript = document.createElement("script");
|
|
if (oldScript.src) {
|
|
newScript.src = oldScript.src;
|
|
} else {
|
|
// Wrap script body so it has access to data and bridge
|
|
newScript.textContent =
|
|
"(function(data, mcp) {\n" +
|
|
oldScript.textContent +
|
|
"\n})(window.__VISUAL_DATA__, window.__MCP__);";
|
|
}
|
|
oldScript.parentNode.replaceChild(newScript, oldScript);
|
|
});
|
|
} else {
|
|
// --- Plain JS path: execute as a script that manipulates #root ---
|
|
const fn = new Function("data", "mcp", "root", "Chart", "L", code);
|
|
fn(data, mcpBridge, root, window.Chart, window.L);
|
|
}
|
|
|
|
// Auto-resize after rendering
|
|
requestAnimationFrame(() => {
|
|
const height = Math.max(root.scrollHeight + 32, 100);
|
|
mcpBridge.resize(height);
|
|
});
|
|
|
|
} catch (err) {
|
|
root.innerHTML =
|
|
'<div class="visual-error">Visual render error:\n' +
|
|
err.message.replace(/</g, "<") +
|
|
"</div>";
|
|
mcpBridge.resize(120);
|
|
}
|
|
}
|
|
|
|
// --- Listen for messages from the host ---
|
|
|
|
window.addEventListener("message", (e) => {
|
|
const msg = e.data;
|
|
if (!msg || typeof msg !== "object") return;
|
|
|
|
// Response to a tools/call we made
|
|
if (msg.id && mcpBridge._pending.has(msg.id)) {
|
|
mcpBridge._pending.get(msg.id)(msg.result);
|
|
mcpBridge._pending.delete(msg.id);
|
|
return;
|
|
}
|
|
|
|
// Host sends visual code to render
|
|
if (msg.method === "render") {
|
|
executeVisual(msg.params?.code || "", msg.params?.data || {});
|
|
return;
|
|
}
|
|
|
|
// Legacy: tool/input for backwards compatibility with MCP App protocol
|
|
if (msg.method === "tool/input") {
|
|
// If params has a `code` field, treat as a visual render
|
|
if (msg.params?.code) {
|
|
executeVisual(msg.params.code, msg.params.data || {});
|
|
}
|
|
}
|
|
});
|
|
|
|
// Tell the host we're ready
|
|
mcpBridge.sendToHost({
|
|
jsonrpc: "2.0",
|
|
method: "ui/initialize",
|
|
params: { runtime: "visual-artifact", version: "1.0.0" },
|
|
});
|
|
</script>
|
|
</body>
|
|
</html>
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add bundle/apps/runtime.html && git commit -m "feat: visual artifact runtime for AI-generated inline visualizations"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: Visual artifacts agent context file
|
|
|
|
This context file teaches the researcher agent HOW to generate inline visuals — Claude "builds visuals" style. The agent generates arbitrary React/HTML/JS code on the fly, not from fixed templates. The context includes when to generate visuals, what libraries are available, the output format, 5 complete working example patterns, and constraints.
|
|
|
|
**Files:**
|
|
- Create: `bundle/context/visual-artifacts.md`
|
|
|
|
**Step 1: Create the visual artifacts context file**
|
|
|
|
Create `bundle/context/visual-artifacts.md`:
|
|
|
|
````markdown
|
|
# Inline Visual Artifacts
|
|
|
|
You can generate interactive visualizations inline in your messages by writing
|
|
code in a `visual` fenced code block. The frontend renders the code live in
|
|
a sandboxed iframe with pre-loaded libraries.
|
|
|
|
## When to Generate a Visual
|
|
|
|
Be proactive. Generate a visual when:
|
|
- **Comparing options**: product comparison tables, feature matrices, pros/cons
|
|
- **Showing trends**: price history, ratings over time, market data
|
|
- **Displaying locations**: dealer maps, store finders, event venues
|
|
- **Scoring/ranking**: decision matrices, weighted scoring tables
|
|
- **Budget breakdowns**: cost comparisons, price distributions
|
|
- **Dashboards**: multi-metric summaries with charts + stats + tables
|
|
|
|
Don't wait to be asked. If you have structured data that would benefit from
|
|
a visual, generate one.
|
|
|
|
## Available Libraries (pre-loaded in the runtime)
|
|
|
|
These are loaded and ready to use — no imports needed:
|
|
|
|
| Library | Global | Version | Use for |
|
|
|---------|--------|---------|---------|
|
|
| **Tabler CSS** | CSS classes | 1.0.0-beta20 | Cards, tables, badges, buttons, forms, grid, dark theme |
|
|
| **Chart.js** | `Chart` | 4.x | Bar, line, pie, radar, doughnut, scatter, polar area charts |
|
|
| **Leaflet.js** | `L` | 1.9 | Maps with markers, popups, tile layers (CARTO dark tiles) |
|
|
| **React** | `React` | 19 | Interactive components with state |
|
|
| **ReactDOM** | `ReactDOM` | 19 | Rendering React components |
|
|
| **Babel** | (automatic) | 7.x | JSX compilation (used automatically when JSX detected) |
|
|
|
|
## Output Format
|
|
|
|
Write the visual as a fenced code block with the `visual` language tag,
|
|
containing a JSON object with `title`, `code`, and optionally `data` and `height`:
|
|
|
|
~~~
|
|
```visual
|
|
{
|
|
"title": "Price Comparison",
|
|
"code": "<your complete HTML, JSX, or JS code here>",
|
|
"data": { "items": [...] },
|
|
"height": 400
|
|
}
|
|
```
|
|
~~~
|
|
|
|
### Fields
|
|
|
|
| Field | Required | Description |
|
|
|-------|----------|-------------|
|
|
| `title` | Yes | Display title shown above the visual |
|
|
| `code` | Yes | The complete code to execute (HTML, JSX, or plain JS) |
|
|
| `data` | No | Structured data object passed to the code as `data` |
|
|
| `height` | No | Initial iframe height in pixels (default: 300, max: 800) |
|
|
|
|
### Code Formats
|
|
|
|
The runtime auto-detects the code format:
|
|
|
|
1. **React/JSX** (contains `<ComponentName` or `React.createElement`): Compiled with Babel, rendered to `#root`. Define an `App` component and it auto-renders.
|
|
2. **HTML** (contains `<div`, `<table`, etc.): Injected directly into `#root`. Script tags execute with `data` and `mcp` in scope.
|
|
3. **Plain JS** (anything else): Executed as a function with `data`, `mcp`, `root`, `Chart`, `L` as parameters.
|
|
|
|
### Available in Code Scope
|
|
|
|
| Variable | Type | Description |
|
|
|----------|------|-------------|
|
|
| `data` | Object | The `data` field from the visual JSON block |
|
|
| `mcp` | Object | MCP bridge: `mcp.sendMessage(text)`, `mcp.callTool(name, args)`, `mcp.resize(height)` |
|
|
| `root` | Element | The `#root` DOM element (for plain JS path) |
|
|
| `Chart` | Class | Chart.js constructor (for plain JS path) |
|
|
| `L` | Object | Leaflet.js namespace (for plain JS path) |
|
|
|
|
## Example Patterns
|
|
|
|
### 1. Comparison Table (HTML + Tabler CSS)
|
|
|
|
```visual
|
|
{
|
|
"title": "Hybrid Cars Under $10K",
|
|
"code": "<div class=\"card\"><div class=\"table-responsive\"><table class=\"table table-hover table-striped\"><thead><tr><th>Model</th><th>Year</th><th>Price</th><th>Miles</th><th>MPG</th><th>Source</th></tr></thead><tbody id=\"rows\"></tbody></table></div></div><script>const rows = data.items.map(item => `<tr><td><strong>${item.model}</strong></td><td>${item.year}</td><td style=\"color:#7af7a2;font-weight:600\">${item.price}</td><td>${item.miles}</td><td>${item.mpg}</td><td><a href=\"${item.url}\" style=\"color:#7aa2f7\" target=\"_blank\">${item.source}</a></td></tr>`).join('');document.getElementById('rows').innerHTML = rows;</script>",
|
|
"data": {
|
|
"items": [
|
|
{"model": "Toyota Prius", "year": 2015, "price": "$7,800", "miles": "82K", "mpg": "51", "source": "CarGurus", "url": "#"},
|
|
{"model": "Honda Civic Hybrid", "year": 2014, "price": "$6,200", "miles": "95K", "mpg": "44", "source": "Craigslist", "url": "#"},
|
|
{"model": "Toyota Camry Hybrid", "year": 2013, "price": "$8,500", "miles": "78K", "mpg": "41", "source": "AutoTrader", "url": "#"},
|
|
{"model": "Ford Fusion Hybrid", "year": 2014, "price": "$5,900", "miles": "110K", "mpg": "42", "source": "Cars.com", "url": "#"}
|
|
]
|
|
},
|
|
"height": 280
|
|
}
|
|
```
|
|
|
|
### 2. Bar Chart (Plain JS + Chart.js)
|
|
|
|
```visual
|
|
{
|
|
"title": "Price Comparison by Model",
|
|
"code": "const canvas = document.createElement('canvas'); root.appendChild(canvas); new Chart(canvas.getContext('2d'), { type: 'bar', data: { labels: data.labels, datasets: [{ label: data.datasetLabel, data: data.values, backgroundColor: 'rgba(122,162,247,0.6)', borderColor: 'rgba(122,162,247,1)', borderWidth: 1 }] }, options: { responsive: true, plugins: { legend: { labels: { color: '#c0c0d0' } }, title: { display: true, text: data.title, color: '#e0e0e0' } }, scales: { x: { ticks: { color: '#8888aa' }, grid: { color: '#1e2a4a' } }, y: { ticks: { color: '#8888aa' }, grid: { color: '#1e2a4a' } } }, onClick: (_, els) => { if (els.length) mcp.sendMessage('Tell me more about ' + data.labels[els[0].index]); } } });",
|
|
"data": {
|
|
"title": "Asking Price ($)",
|
|
"datasetLabel": "Price",
|
|
"labels": ["Prius", "Civic Hybrid", "Camry Hybrid", "Fusion Hybrid", "Insight"],
|
|
"values": [7800, 6200, 8500, 5900, 4800]
|
|
},
|
|
"height": 380
|
|
}
|
|
```
|
|
|
|
### 3. Location Map (Plain JS + Leaflet)
|
|
|
|
```visual
|
|
{
|
|
"title": "Dealers Near Woodinville, WA",
|
|
"code": "root.style.height = '340px'; const map = L.map(root).setView(data.center, data.zoom); L.tileLayer('https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}{r}.png', { attribution: 'CARTO', maxZoom: 18 }).addTo(map); data.markers.forEach(m => { const marker = L.marker([m.lat, m.lng]).addTo(map); marker.bindPopup(`<b>${m.title}</b><br>${m.description || ''}<br><span style='color:#7af7a2'>${m.price || ''}</span>`); marker.on('click', () => mcp.sendMessage('Tell me about ' + m.title)); }); if (data.markers.length > 1) map.fitBounds(data.markers.map(m => [m.lat, m.lng]), { padding: [30, 30] });",
|
|
"data": {
|
|
"center": [47.76, -122.16],
|
|
"zoom": 11,
|
|
"markers": [
|
|
{"lat": 47.758, "lng": -122.164, "title": "Toyota of Kirkland", "description": "Certified pre-owned", "price": "$7,800"},
|
|
{"lat": 47.81, "lng": -122.20, "title": "Honda of Bothell", "description": "Used inventory", "price": "$6,200"},
|
|
{"lat": 47.68, "lng": -122.12, "title": "CarMax Bellevue", "description": "Multi-brand", "price": "Various"}
|
|
]
|
|
},
|
|
"height": 380
|
|
}
|
|
```
|
|
|
|
### 4. React Dashboard (JSX)
|
|
|
|
```visual
|
|
{
|
|
"title": "Research Summary Dashboard",
|
|
"code": "function StatCard({ label, value, color }) { return React.createElement('div', { className: 'card', style: { flex: 1, minWidth: 140 } }, React.createElement('div', { className: 'card-body', style: { padding: '1rem' } }, React.createElement('div', { style: { fontSize: '0.75rem', color: '#8888aa', textTransform: 'uppercase' } }, label), React.createElement('div', { style: { fontSize: '1.5rem', fontWeight: 700, color: color || '#e0e0e0' } }, value))); } function App({ data }) { const [sort, setSort] = React.useState('price'); const sorted = [...data.items].sort((a, b) => sort === 'price' ? a.priceNum - b.priceNum : a.miles - b.miles); return React.createElement('div', null, React.createElement('div', { style: { display: 'flex', gap: '0.5rem', marginBottom: '1rem' } }, React.createElement(StatCard, { label: 'Options Found', value: data.items.length, color: '#7aa2f7' }), React.createElement(StatCard, { label: 'Avg Price', value: '$' + Math.round(data.items.reduce((s, i) => s + i.priceNum, 0) / data.items.length).toLocaleString(), color: '#7af7a2' }), React.createElement(StatCard, { label: 'Price Range', value: '$' + Math.min(...data.items.map(i => i.priceNum)).toLocaleString() + ' - $' + Math.max(...data.items.map(i => i.priceNum)).toLocaleString() })), React.createElement('div', { style: { marginBottom: '0.5rem' } }, React.createElement('button', { onClick: () => setSort('price'), className: 'btn btn-sm ' + (sort === 'price' ? 'btn-primary' : 'btn-outline-secondary'), style: { marginRight: 4 } }, 'Sort by Price'), React.createElement('button', { onClick: () => setSort('miles'), className: 'btn btn-sm ' + (sort === 'miles' ? 'btn-primary' : 'btn-outline-secondary') }, 'Sort by Miles')), React.createElement('div', { className: 'table-responsive' }, React.createElement('table', { className: 'table table-hover' }, React.createElement('thead', null, React.createElement('tr', null, ['Model', 'Price', 'Miles', 'Source'].map(h => React.createElement('th', { key: h }, h)))), React.createElement('tbody', null, sorted.map((item, i) => React.createElement('tr', { key: i, onClick: () => mcp.sendMessage('Tell me more about ' + item.model), style: { cursor: 'pointer' } }, React.createElement('td', null, React.createElement('strong', null, item.model)), React.createElement('td', { style: { color: '#7af7a2' } }, item.price), React.createElement('td', null, item.milesStr), React.createElement('td', null, item.source))))))); }",
|
|
"data": {
|
|
"items": [
|
|
{"model": "2015 Prius", "price": "$7,800", "priceNum": 7800, "miles": 82000, "milesStr": "82K", "source": "CarGurus"},
|
|
{"model": "2014 Civic Hybrid", "price": "$6,200", "priceNum": 6200, "miles": 95000, "milesStr": "95K", "source": "Craigslist"},
|
|
{"model": "2013 Camry Hybrid", "price": "$8,500", "priceNum": 8500, "miles": 78000, "milesStr": "78K", "source": "AutoTrader"},
|
|
{"model": "2014 Fusion Hybrid", "price": "$5,900", "priceNum": 5900, "miles": 110000, "milesStr": "110K", "source": "Cars.com"}
|
|
]
|
|
},
|
|
"height": 450
|
|
}
|
|
```
|
|
|
|
### 5. Decision Matrix (HTML + interactive weight sliders)
|
|
|
|
```visual
|
|
{
|
|
"title": "Car Decision Matrix",
|
|
"code": "<style>.matrix-table th, .matrix-table td { padding: 0.4rem 0.6rem; text-align: center; } .matrix-table th { color: #8888aa; font-size: 0.75rem; text-transform: uppercase; } .score-cell { font-weight: 600; } .best { color: #7af7a2; } .weight-slider { width: 80px; accent-color: #7aa2f7; }</style><div class=\"card\"><div class=\"card-body\"><div id=\"weights-row\" style=\"display:flex;gap:1rem;margin-bottom:1rem;flex-wrap:wrap\"></div><div class=\"table-responsive\"><table class=\"table table-hover matrix-table\" id=\"matrix\"></table></div></div></div><script>const criteria = data.criteria; const options = data.options; let weights = criteria.map(c => c.weight); function render() { const total = weights.reduce((s,w) => s+w, 0) || 1; const norm = weights.map(w => w/total); let html = '<thead><tr><th>Option</th>'; criteria.forEach((c,i) => html += `<th>${c.name} (${Math.round(norm[i]*100)}%)</th>`); html += '<th>Score</th></tr></thead><tbody>'; const scores = options.map(opt => { let s = 0; opt.scores.forEach((v,i) => s += v * norm[i]); return s; }); const maxScore = Math.max(...scores); options.forEach((opt,oi) => { const isMax = scores[oi] === maxScore; html += `<tr><td style=\"text-align:left\"><strong>${opt.name}</strong></td>`; opt.scores.forEach(s => html += `<td class=\"score-cell\">${s}/10</td>`); html += `<td class=\"score-cell ${isMax ? 'best' : ''}\" style=\"font-size:1.1rem\">${scores[oi].toFixed(1)}</td></tr>`; }); html += '</tbody>'; document.getElementById('matrix').innerHTML = html; } const wr = document.getElementById('weights-row'); criteria.forEach((c,i) => { const label = document.createElement('label'); label.style.cssText = 'font-size:0.75rem;color:#aaa;display:flex;flex-direction:column;align-items:center'; label.textContent = c.name; const slider = document.createElement('input'); slider.type = 'range'; slider.min = 0; slider.max = 10; slider.value = weights[i]; slider.className = 'weight-slider'; slider.oninput = () => { weights[i] = +slider.value; render(); }; label.appendChild(slider); wr.appendChild(label); }); render();</script>",
|
|
"data": {
|
|
"criteria": [
|
|
{"name": "Price", "weight": 8},
|
|
{"name": "MPG", "weight": 7},
|
|
{"name": "Reliability", "weight": 9},
|
|
{"name": "Cargo", "weight": 4},
|
|
{"name": "Comfort", "weight": 5}
|
|
],
|
|
"options": [
|
|
{"name": "2015 Prius", "scores": [7, 10, 9, 5, 6]},
|
|
{"name": "2014 Civic Hybrid", "scores": [8, 8, 8, 6, 7]},
|
|
{"name": "2013 Camry Hybrid", "scores": [6, 7, 9, 8, 9]},
|
|
{"name": "2014 Fusion Hybrid", "scores": [9, 7, 6, 7, 8]}
|
|
]
|
|
},
|
|
"height": 400
|
|
}
|
|
```
|
|
|
|
## Constraints
|
|
|
|
- **Self-contained**: All code must work without external API calls or fetches.
|
|
Data comes from the `data` field or is hardcoded in the code.
|
|
- **Dark theme always**: Use Tabler's `.theme-dark` classes. Background is
|
|
transparent (inherits from runtime). Text should be light (#e0e0e0).
|
|
- **Responsive width**: Visuals fill the available width. Use percentage widths
|
|
and `max-width` where appropriate.
|
|
- **Max height**: Specify `height` in the JSON block (default 300, max 800).
|
|
The runtime auto-resizes based on content.
|
|
- **No external fetches**: Don't call APIs from within the visual. All data
|
|
must be passed via the `data` field.
|
|
- **Click-to-explore**: When users click interactive elements (chart bars,
|
|
table rows, map markers), call `mcp.sendMessage("Tell me more about X")`
|
|
to inject a follow-up question into the chat.
|
|
|
|
## When to Use Visuals vs A2UI
|
|
|
|
| Need | Use |
|
|
|------|-----|
|
|
| Toggle filters, simple form inputs | A2UI blocks |
|
|
| Static data table (< 5 rows) | A2UI table block |
|
|
| Interactive chart with click events | `visual` block |
|
|
| Sortable table with many rows | `visual` block |
|
|
| Side-by-side product comparison | `visual` block |
|
|
| Location map with markers | `visual` block |
|
|
| Multi-metric dashboard | `visual` block |
|
|
| Decision matrix with adjustable weights | `visual` block |
|
|
````
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add bundle/context/visual-artifacts.md && git commit -m "feat: visual artifacts context file with 5 example patterns for AI-generated visuals"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: Verify bundle structure and visual artifact references
|
|
|
|
The researcher agent definition references `@mention bundle/context/visual-artifacts.md` (from Task 3). This task verifies the agent can be loaded, the bundle structure is valid, and all @mention references resolve.
|
|
|
|
**Files:**
|
|
- Verify: `bundle/agents/researcher.md` (updated in Task 3 with inline visuals section)
|
|
- Verify: `bundle/apps/runtime.html` (created in Task 5)
|
|
- Verify: `bundle/context/visual-artifacts.md` (created in Task 6)
|
|
|
|
**Step 1: Verify bundle directory structure**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
find bundle/ -type f | sort
|
|
```
|
|
|
|
Expected output:
|
|
```
|
|
bundle/agents/researcher.md
|
|
bundle/apps/runtime.html
|
|
bundle/behaviors/research-workbench.yaml
|
|
bundle/bundle.md
|
|
bundle/context/a2ui-format.md
|
|
bundle/context/artifact-conventions.md
|
|
bundle/context/researcher-instructions.md
|
|
bundle/context/visual-artifacts.md
|
|
```
|
|
|
|
**Step 2: Verify all @mention references resolve**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
# Check that all @mention targets exist
|
|
grep -rh '@mention' bundle/ | sed 's/.*@mention //' | while read ref; do
|
|
# Convert bundle reference to file path
|
|
path="bundle/$(echo "$ref" | sed 's|^bundle/||').md"
|
|
if [ -f "$path" ]; then
|
|
echo "OK: $path"
|
|
else
|
|
echo "MISSING: $path (from @mention $ref)"
|
|
fi
|
|
done
|
|
```
|
|
|
|
Expected: All references resolve to OK. Specifically:
|
|
- `bundle/context/a2ui-format.md` — OK
|
|
- `bundle/context/artifact-conventions.md` — OK
|
|
- `bundle/context/visual-artifacts.md` — OK
|
|
- `bundle/context/researcher-instructions.md` — OK (via behavior include)
|
|
|
|
**Step 3: Verify runtime.html loads the expected libraries**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
# Quick sanity check that runtime.html references all expected CDN libraries
|
|
grep -c 'chart.js' bundle/apps/runtime.html && \
|
|
grep -c 'leaflet' bundle/apps/runtime.html && \
|
|
grep -c 'react@19' bundle/apps/runtime.html && \
|
|
grep -c 'babel/standalone' bundle/apps/runtime.html && \
|
|
echo "All libraries referenced in runtime.html"
|
|
```
|
|
|
|
Expected: 4 counts (one per library family) and "All libraries referenced".
|
|
|
|
**Step 4: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add bundle/ && git commit -m "feat: complete bundle with visual artifact runtime and context"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 8: Browser panel component (noVNC iframe)
|
|
|
|
**Files:**
|
|
- Create: `frontend/src/components/BrowserPanel.tsx`
|
|
|
|
**Step 1: Create the browser panel**
|
|
|
|
Create `frontend/src/components/BrowserPanel.tsx`:
|
|
|
|
```tsx
|
|
/**
|
|
* Browser panel: embeds a noVNC iframe showing the live Playwright browser.
|
|
*
|
|
* In production (behind cloudflared): points to vnc.ampbox.io
|
|
* In local dev (Docker): points to localhost:6080
|
|
* The URL is determined by the current hostname.
|
|
*/
|
|
|
|
interface BrowserPanelProps {
|
|
sessionId: string | null;
|
|
}
|
|
|
|
function getVncUrl(): string {
|
|
const hostname = window.location.hostname;
|
|
|
|
// Production: behind cloudflared
|
|
if (hostname === "research.ampbox.io") {
|
|
return "https://vnc.ampbox.io/vnc.html?autoconnect=true&resize=remote&quality=6&compression=2";
|
|
}
|
|
|
|
// Local Docker: noVNC on port 6080
|
|
return `http://${hostname}:6080/vnc.html?autoconnect=true&resize=remote&quality=6&compression=2`;
|
|
}
|
|
|
|
export function BrowserPanel({ sessionId }: BrowserPanelProps) {
|
|
if (!sessionId) {
|
|
return (
|
|
<div style={{ padding: "2rem", color: "#6a6a8a", textAlign: "center" }}>
|
|
Select a session to view the browser
|
|
</div>
|
|
);
|
|
}
|
|
|
|
return (
|
|
<iframe
|
|
src={getVncUrl()}
|
|
title="Browser - noVNC"
|
|
style={{
|
|
width: "100%",
|
|
height: "100%",
|
|
border: "none",
|
|
backgroundColor: "#0a0a1a",
|
|
}}
|
|
allow="clipboard-read; clipboard-write"
|
|
/>
|
|
);
|
|
}
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add frontend/src/components/BrowserPanel.tsx && git commit -m "feat: browser panel with noVNC iframe"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 9: Artifacts panel component
|
|
|
|
**Files:**
|
|
- Create: `frontend/src/components/ArtifactsPanel.tsx`
|
|
|
|
**Step 1: Create the artifacts panel**
|
|
|
|
Create `frontend/src/components/ArtifactsPanel.tsx`:
|
|
|
|
```tsx
|
|
import { useEffect } from "react";
|
|
import ReactMarkdown from "react-markdown";
|
|
import { IconFileText, IconRefresh } from "@tabler/icons-react";
|
|
import { useArtifacts } from "../hooks/useArtifacts";
|
|
|
|
interface ArtifactsPanelProps {
|
|
sessionId: string | null;
|
|
}
|
|
|
|
export function ArtifactsPanel({ sessionId }: ArtifactsPanelProps) {
|
|
const { artifacts, activeArtifact, refresh, loadArtifact } = useArtifacts(sessionId);
|
|
|
|
// Refresh artifact list when session changes
|
|
useEffect(() => {
|
|
if (sessionId) refresh();
|
|
}, [sessionId, refresh]);
|
|
|
|
if (!sessionId) {
|
|
return (
|
|
<div style={{ padding: "2rem", color: "#6a6a8a", textAlign: "center" }}>
|
|
Select a session to view artifacts
|
|
</div>
|
|
);
|
|
}
|
|
|
|
return (
|
|
<div style={{ display: "flex", flexDirection: "column", height: "100%" }}>
|
|
{/* Artifact list header */}
|
|
<div
|
|
style={{
|
|
padding: "0.5rem 0.75rem",
|
|
borderBottom: "1px solid #2a2a4a",
|
|
display: "flex",
|
|
justifyContent: "space-between",
|
|
alignItems: "center",
|
|
}}
|
|
>
|
|
<span style={{ fontSize: "0.8rem", color: "#8888aa" }}>
|
|
{artifacts.length} artifact{artifacts.length !== 1 ? "s" : ""}
|
|
</span>
|
|
<button
|
|
onClick={refresh}
|
|
style={{
|
|
background: "none",
|
|
border: "none",
|
|
color: "#7aa2f7",
|
|
cursor: "pointer",
|
|
padding: "2px",
|
|
}}
|
|
title="Refresh"
|
|
>
|
|
<IconRefresh size={16} />
|
|
</button>
|
|
</div>
|
|
|
|
{/* Artifact tabs */}
|
|
{artifacts.length > 0 && (
|
|
<div
|
|
style={{
|
|
display: "flex",
|
|
gap: "2px",
|
|
padding: "0.3rem 0.5rem",
|
|
overflowX: "auto",
|
|
borderBottom: "1px solid #2a2a4a",
|
|
}}
|
|
>
|
|
{artifacts.map((a) => (
|
|
<button
|
|
key={a.name}
|
|
onClick={() => loadArtifact(a.name)}
|
|
style={{
|
|
background:
|
|
activeArtifact?.name === a.name ? "#2a3a5a" : "transparent",
|
|
border: "none",
|
|
color: activeArtifact?.name === a.name ? "#ffffff" : "#8888aa",
|
|
padding: "0.3rem 0.6rem",
|
|
borderRadius: "4px",
|
|
cursor: "pointer",
|
|
fontSize: "0.75rem",
|
|
whiteSpace: "nowrap",
|
|
display: "flex",
|
|
alignItems: "center",
|
|
gap: "4px",
|
|
}}
|
|
>
|
|
<IconFileText size={14} />
|
|
{a.name}
|
|
</button>
|
|
))}
|
|
</div>
|
|
)}
|
|
|
|
{/* Artifact content */}
|
|
<div
|
|
style={{
|
|
flex: 1,
|
|
overflowY: "auto",
|
|
padding: "1rem",
|
|
}}
|
|
>
|
|
{activeArtifact ? (
|
|
<div className="chat-message assistant">
|
|
<ReactMarkdown>{activeArtifact.content}</ReactMarkdown>
|
|
</div>
|
|
) : artifacts.length > 0 ? (
|
|
<div style={{ color: "#6a6a8a", textAlign: "center", paddingTop: "2rem" }}>
|
|
Click an artifact tab to view it
|
|
</div>
|
|
) : (
|
|
<div style={{ color: "#6a6a8a", textAlign: "center", paddingTop: "2rem" }}>
|
|
No artifacts yet. The researcher will create them during research.
|
|
</div>
|
|
)}
|
|
</div>
|
|
</div>
|
|
);
|
|
}
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add frontend/src/components/ArtifactsPanel.tsx && git commit -m "feat: artifacts panel with tabbed markdown rendering"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 10: A2UI renderer component
|
|
|
|
Parses `a2ui` JSON code blocks from assistant messages and renders them as interactive Tabler components.
|
|
|
|
**Files:**
|
|
- Create: `frontend/src/components/A2UIRenderer.tsx`
|
|
|
|
**Step 1: Create the A2UI renderer**
|
|
|
|
Create `frontend/src/components/A2UIRenderer.tsx`:
|
|
|
|
```tsx
|
|
import type { A2UIBlock } from "../lib/types";
|
|
import { IconFilter, IconTable, IconCards, IconForms } from "@tabler/icons-react";
|
|
|
|
interface A2UIRendererProps {
|
|
block: A2UIBlock;
|
|
onAction?: (action: string, data: Record<string, unknown>) => void;
|
|
}
|
|
|
|
export function A2UIRenderer({ block, onAction }: A2UIRendererProps) {
|
|
switch (block.type) {
|
|
case "filters":
|
|
return (
|
|
<div className="tool-call-card" style={{ display: "flex", gap: "0.4rem", flexWrap: "wrap", alignItems: "center" }}>
|
|
<IconFilter size={16} style={{ color: "#7aa2f7", marginRight: 4 }} />
|
|
{block.options.map((opt, i) => (
|
|
<button
|
|
key={i}
|
|
onClick={() => onAction?.("filter_toggle", { label: opt.label, active: !opt.active })}
|
|
style={{
|
|
padding: "0.3rem 0.6rem",
|
|
borderRadius: "12px",
|
|
border: "1px solid",
|
|
borderColor: opt.active ? "#5a7aba" : "#3a3a5a",
|
|
backgroundColor: opt.active ? "#2a3a6a" : "transparent",
|
|
color: opt.active ? "#ffffff" : "#8888aa",
|
|
cursor: "pointer",
|
|
fontSize: "0.8rem",
|
|
}}
|
|
>
|
|
{opt.label}
|
|
</button>
|
|
))}
|
|
</div>
|
|
);
|
|
|
|
case "table":
|
|
return (
|
|
<div className="tool-call-card" style={{ overflowX: "auto" }}>
|
|
<div style={{ marginBottom: "0.4rem", display: "flex", alignItems: "center", gap: 4 }}>
|
|
<IconTable size={16} style={{ color: "#7aa2f7" }} />
|
|
<span style={{ fontSize: "0.8rem", color: "#8888aa" }}>Data Table</span>
|
|
</div>
|
|
<table style={{ width: "100%", borderCollapse: "collapse", fontSize: "0.8rem" }}>
|
|
<thead>
|
|
<tr>
|
|
{block.headers.map((h, i) => (
|
|
<th
|
|
key={i}
|
|
style={{
|
|
textAlign: "left",
|
|
padding: "0.3rem 0.5rem",
|
|
borderBottom: "1px solid #2a2a4a",
|
|
color: "#aaaacc",
|
|
fontWeight: 600,
|
|
}}
|
|
>
|
|
{h}
|
|
</th>
|
|
))}
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
{block.rows.map((row, ri) => (
|
|
<tr key={ri}>
|
|
{row.map((cell, ci) => (
|
|
<td
|
|
key={ci}
|
|
style={{
|
|
padding: "0.3rem 0.5rem",
|
|
borderBottom: "1px solid #1a1a2e",
|
|
color: "#c0c0d0",
|
|
}}
|
|
>
|
|
{cell}
|
|
</td>
|
|
))}
|
|
</tr>
|
|
))}
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
);
|
|
|
|
case "cards":
|
|
return (
|
|
<div style={{ display: "grid", gridTemplateColumns: "repeat(auto-fill, minmax(200px, 1fr))", gap: "0.5rem", margin: "0.5rem 0" }}>
|
|
{block.items.map((item, i) => (
|
|
<div
|
|
key={i}
|
|
className="tool-call-card"
|
|
style={{ cursor: item.url ? "pointer" : "default" }}
|
|
onClick={() => item.url && window.open(item.url, "_blank")}
|
|
>
|
|
<div style={{ fontWeight: 600, color: "#e0e0e0", marginBottom: 4 }}>{item.title}</div>
|
|
{item.price && (
|
|
<div style={{ color: "#7af7a2", fontWeight: 600, fontSize: "0.9rem" }}>{item.price}</div>
|
|
)}
|
|
{item.description && (
|
|
<div style={{ color: "#8888aa", fontSize: "0.75rem", marginTop: 4 }}>{item.description}</div>
|
|
)}
|
|
</div>
|
|
))}
|
|
</div>
|
|
);
|
|
|
|
case "form":
|
|
return (
|
|
<div className="tool-call-card">
|
|
<div style={{ marginBottom: "0.5rem", display: "flex", alignItems: "center", gap: 4 }}>
|
|
<IconForms size={16} style={{ color: "#7aa2f7" }} />
|
|
<span style={{ fontSize: "0.8rem", color: "#8888aa" }}>Input Form</span>
|
|
</div>
|
|
<form
|
|
onSubmit={(e) => {
|
|
e.preventDefault();
|
|
const formData = new FormData(e.currentTarget);
|
|
const data: Record<string, unknown> = {};
|
|
formData.forEach((v, k) => (data[k] = v));
|
|
onAction?.("form_submit", data);
|
|
}}
|
|
style={{ display: "flex", flexDirection: "column", gap: "0.4rem" }}
|
|
>
|
|
{block.fields.map((field) => (
|
|
<div key={field.name}>
|
|
<label style={{ fontSize: "0.75rem", color: "#8888aa", display: "block", marginBottom: 2 }}>
|
|
{field.label || field.name}
|
|
</label>
|
|
{field.type === "select" ? (
|
|
<select
|
|
name={field.name}
|
|
defaultValue={String(field.value ?? "")}
|
|
style={{
|
|
width: "100%",
|
|
padding: "0.3rem",
|
|
backgroundColor: "#1e2a4a",
|
|
border: "1px solid #3a3a5a",
|
|
borderRadius: "4px",
|
|
color: "#e0e0e0",
|
|
fontSize: "0.8rem",
|
|
}}
|
|
>
|
|
{field.options?.map((opt) => (
|
|
<option key={opt} value={opt}>{opt}</option>
|
|
))}
|
|
</select>
|
|
) : (
|
|
<input
|
|
name={field.name}
|
|
type={field.type}
|
|
defaultValue={String(field.value ?? "")}
|
|
style={{
|
|
width: "100%",
|
|
padding: "0.3rem",
|
|
backgroundColor: "#1e2a4a",
|
|
border: "1px solid #3a3a5a",
|
|
borderRadius: "4px",
|
|
color: "#e0e0e0",
|
|
fontSize: "0.8rem",
|
|
boxSizing: "border-box",
|
|
}}
|
|
/>
|
|
)}
|
|
</div>
|
|
))}
|
|
<button
|
|
type="submit"
|
|
style={{
|
|
padding: "0.4rem 0.8rem",
|
|
backgroundColor: "#3a5aba",
|
|
color: "white",
|
|
border: "none",
|
|
borderRadius: "4px",
|
|
cursor: "pointer",
|
|
fontSize: "0.8rem",
|
|
marginTop: "0.3rem",
|
|
alignSelf: "flex-start",
|
|
}}
|
|
>
|
|
{block.submitLabel || "Submit"}
|
|
</button>
|
|
</form>
|
|
</div>
|
|
);
|
|
|
|
default:
|
|
return null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Parse A2UI blocks from message content.
|
|
* Looks for ```a2ui ... ``` fenced code blocks and extracts the JSON.
|
|
*/
|
|
export function parseA2UIBlocks(content: string): A2UIBlock[] {
|
|
const blocks: A2UIBlock[] = [];
|
|
const regex = /```a2ui\s*\n([\s\S]*?)```/g;
|
|
let match;
|
|
while ((match = regex.exec(content)) !== null) {
|
|
try {
|
|
const parsed = JSON.parse(match[1].trim());
|
|
if (parsed && parsed.type) {
|
|
blocks.push(parsed as A2UIBlock);
|
|
}
|
|
} catch {
|
|
// Skip unparseable blocks
|
|
}
|
|
}
|
|
return blocks;
|
|
}
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add frontend/src/components/A2UIRenderer.tsx && git commit -m "feat: A2UI renderer for filters, tables, cards, forms"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 11: Wire up RightPanel and integrate A2UI + visual blocks into ChatPanel
|
|
|
|
**Files:**
|
|
- Modify: `frontend/src/components/RightPanel.tsx`
|
|
- Modify: `frontend/src/components/ChatPanel.tsx`
|
|
|
|
**Step 1: Update RightPanel with real Browser and Artifacts components**
|
|
|
|
Replace `frontend/src/components/RightPanel.tsx` with:
|
|
|
|
```tsx
|
|
import { useState } from "react";
|
|
import { BrowserPanel } from "./BrowserPanel";
|
|
import { ArtifactsPanel } from "./ArtifactsPanel";
|
|
|
|
interface RightPanelProps {
|
|
sessionId: string | null;
|
|
}
|
|
|
|
export function RightPanel({ sessionId }: RightPanelProps) {
|
|
const [activeTab, setActiveTab] = useState<"browser" | "artifacts">("browser");
|
|
|
|
return (
|
|
<div className="right-panel">
|
|
<div className="right-panel-tabs">
|
|
<button
|
|
className={activeTab === "browser" ? "active" : ""}
|
|
onClick={() => setActiveTab("browser")}
|
|
>
|
|
Browser
|
|
</button>
|
|
<button
|
|
className={activeTab === "artifacts" ? "active" : ""}
|
|
onClick={() => setActiveTab("artifacts")}
|
|
>
|
|
Artifacts
|
|
</button>
|
|
</div>
|
|
<div className="right-panel-content">
|
|
{activeTab === "browser" ? (
|
|
<BrowserPanel sessionId={sessionId} />
|
|
) : (
|
|
<ArtifactsPanel sessionId={sessionId} />
|
|
)}
|
|
</div>
|
|
</div>
|
|
);
|
|
}
|
|
```
|
|
|
|
**Step 2: Update ChatPanel to render A2UI blocks and visual blocks**
|
|
|
|
In `frontend/src/components/ChatPanel.tsx`, add the A2UI import and rendering. Replace the message rendering section inside the `messages.map()` where assistant messages are rendered. Specifically, after the `<ReactMarkdown>` call, add A2UI parsing:
|
|
|
|
Add this import at the top:
|
|
```tsx
|
|
import { A2UIRenderer, parseA2UIBlocks } from "./A2UIRenderer";
|
|
```
|
|
|
|
Then in the assistant message rendering block (inside `messages.map`), replace:
|
|
```tsx
|
|
{msg.role === "assistant" ? (
|
|
<>
|
|
<ReactMarkdown>{msg.content}</ReactMarkdown>
|
|
{msg.toolCalls?.map((tc) => (
|
|
<ToolCallCard key={tc.id} toolCall={tc} />
|
|
))}
|
|
</>
|
|
```
|
|
|
|
With:
|
|
```tsx
|
|
{msg.role === "assistant" ? (
|
|
<>
|
|
<ReactMarkdown>{msg.content}</ReactMarkdown>
|
|
{parseA2UIBlocks(msg.content).map((block, i) => (
|
|
<A2UIRenderer
|
|
key={i}
|
|
block={block}
|
|
onAction={(action, data) => {
|
|
// Send A2UI interaction back as user message
|
|
onSend(`[${action}] ${JSON.stringify(data)}`);
|
|
}}
|
|
/>
|
|
))}
|
|
{msg.toolCalls?.map((tc) => (
|
|
<ToolCallCard key={tc.id} toolCall={tc} />
|
|
))}
|
|
</>
|
|
```
|
|
|
|
**Step 3: Verify the build compiles**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench/frontend
|
|
npm run build 2>&1 | tail -10
|
|
```
|
|
|
|
Expected: Build succeeds.
|
|
|
|
**Step 4: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add frontend/src/components/ && git commit -m "feat: wire RightPanel with Browser + Artifacts, add A2UI rendering to chat"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 12: Docker Compose for local development
|
|
|
|
**Files:**
|
|
- Create: `docker-compose.yml`
|
|
|
|
**Step 1: Create docker-compose.yml**
|
|
|
|
Create `docker-compose.yml`:
|
|
|
|
```yaml
|
|
# Local development: docker compose up
|
|
# Builds and runs the all-in-one container.
|
|
#
|
|
# Before running:
|
|
# 1. Build the frontend: cd frontend && npm run build
|
|
# 2. Set your API key: export ANTHROPIC_API_KEY=sk-...
|
|
# 3. Generate a password hash: python -c "import bcrypt; print(bcrypt.hashpw(b'YOUR_PASSWORD', bcrypt.gensalt()).decode())"
|
|
|
|
services:
|
|
workbench:
|
|
build: .
|
|
ports:
|
|
- "8080:8080" # Web UI
|
|
- "6080:6080" # noVNC browser view
|
|
environment:
|
|
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
|
|
- AUTH_USER=${AUTH_USER:-admin@localhost}
|
|
- AUTH_PASS_HASH=${AUTH_PASS_HASH}
|
|
- DISPLAY=:99
|
|
volumes:
|
|
- ./artifacts:/app/artifacts # Persist artifacts across restarts
|
|
restart: unless-stopped
|
|
```
|
|
|
|
**Step 2: Commit**
|
|
|
|
Run:
|
|
```bash
|
|
cd /home/ken/workspace/research-workbench
|
|
git add docker-compose.yml && git commit -m "feat: docker-compose.yml for local development"
|
|
```
|
|
|
|
---
|
|
|
|
## Cross-Phase Dependencies
|
|
|
|
The visual artifact system introduced in Tasks 5-6 requires coordinating changes across all three phases. These are notes for the implementer:
|
|
|
|
### Phase 1 Backend (already handled)
|
|
|
|
The backend already serves `GET /api/apps/{app_name}` which reads HTML files from `bundle/apps/`. Since we replaced 4 template files with a single `runtime.html`, the endpoint continues to work — the frontend will request `/api/apps/runtime` and get `runtime.html`.
|
|
|
|
The AG-UI adapter already passes through `_meta.ui` and `structuredContent` from tool results. No changes needed.
|
|
|
|
### Phase 2 Frontend (needs updating when implementing Phase 3)
|
|
|
|
The `MCPAppRenderer` component (Phase 2, Task 9) and `ChatPanel` (Phase 2, Tasks 7 + 10) need these updates to support `visual` code blocks:
|
|
|
|
1. **Detect `visual` code blocks in assistant messages**: Add a `parseVisualBlocks(content)` function (similar to `parseA2UIBlocks`) that extracts `` ```visual `` fenced code blocks and parses the JSON (`{ title, code, data, height }`).
|
|
|
|
2. **Create a `VisualBlock` wrapper component** that:
|
|
- Displays the visual's `title` in a header bar
|
|
- Loads `runtime.html` from `/api/apps/runtime` via the existing `MCPAppRenderer`
|
|
- After the iframe signals `ui/initialize`, sends the `render` method via postMessage with the visual's `code` and `data`
|
|
- Supports a "pop out" button to open the visual in a full browser tab
|
|
- Supports a "view source" toggle to show the generated code
|
|
|
|
3. **Wire `VisualBlock` into `ChatPanel`**: After A2UI blocks, render any detected visual blocks from the assistant message content.
|
|
|
|
4. **Update `MCPAppRenderer`**: Add support for the `render` postMessage method (in addition to existing `tool/input`). When the iframe sends `ui/initialize` with `runtime: "visual-artifact"`, send `{ method: "render", params: { code, data } }` instead of `{ method: "tool/input", params: data }`.
|
|
|
|
These updates should be implemented during Phase 3 Task 11 (which already modifies `ChatPanel.tsx`).
|
|
|
|
---
|
|
|
|
## Phase 3 Complete
|
|
|
|
At this point you have:
|
|
- **Bundle**: Thin root including foundation, with researcher agent, A2UI format spec, artifact conventions, and visual artifact generation context
|
|
- **Visual artifact runtime**: Single `runtime.html` that executes arbitrary AI-generated React/HTML/JS code in a sandboxed iframe with pre-loaded Chart.js, Leaflet, React 19, Babel, and Tabler CSS
|
|
- **Visual artifacts context**: Agent instructions with 5 complete example patterns (comparison table, bar chart, location map, React dashboard, decision matrix with interactive weight sliders)
|
|
- **Browser panel**: noVNC iframe with auto-detection of production vs local URLs
|
|
- **Artifacts panel**: Tabbed markdown rendering of per-session artifacts
|
|
- **A2UI renderer**: Interactive filters, tables, cards, and forms from agent-generated JSON
|
|
- **Docker Compose**: One-command local deployment
|
|
|
|
**The AI-generated visuals flow:**
|
|
|
|
```
|
|
Agent writes a ```visual code block in its message
|
|
-> Frontend detects the visual block, parses JSON { title, code, data, height }
|
|
-> VisualBlock component loads runtime.html from /api/apps/runtime
|
|
-> After iframe ready (ui/initialize), sends { method: "render", params: { code, data } }
|
|
-> runtime.html receives code, auto-detects format (JSX/HTML/JS)
|
|
-> Compiles JSX via Babel if needed, executes code with data in scope
|
|
-> Renders interactive chart/table/map/dashboard in the iframe
|
|
-> User clicks elements -> mcp.sendMessage() injects follow-up into chat
|
|
```
|
|
|
|
**Full deployment flow:**
|
|
|
|
```bash
|
|
# 1. Build frontend
|
|
cd frontend && npm run build && cd ..
|
|
|
|
# 2. Build and run Docker container
|
|
export ANTHROPIC_API_KEY=sk-...
|
|
export AUTH_USER=ken@example.com
|
|
export AUTH_PASS_HASH=$(python3 -c "import bcrypt; print(bcrypt.hashpw(b'mypassword', bcrypt.gensalt()).decode())")
|
|
docker compose up --build
|
|
|
|
# 3. Open http://localhost:8080
|
|
# 4. Log in with your credentials
|
|
# 5. Create a session and start researching!
|
|
```
|
|
|
|
**Production deployment (vela0 + cloudflared):**
|
|
|
|
The container exposes ports 8080 and 6080. The existing `/mnt/services/research.yml` cloudflared config routes:
|
|
- `research.ampbox.io` -> `http://10.66.204.209:8080`
|
|
- `vnc.ampbox.io` -> `http://10.66.204.209:6080`
|