feat: visual artifacts context file with 5 example patterns for AI-generated visuals
This commit is contained in:
Ken
@@ -0,0 +1,274 @@
|
||||
# Visual Artifacts (Inline Visuals)
|
||||
|
||||
Visual artifacts let you generate rich, interactive charts, maps, tables, and dashboards
|
||||
directly in the chat. Write a `visual` fenced code block containing JSON with a `code`
|
||||
field, and the frontend renders it live inside a sandboxed iframe using pre-loaded libraries
|
||||
— no installs, no API fetches, no user setup required.
|
||||
|
||||
Use visuals proactively. When data benefits from a chart, table, or map, generate the visual
|
||||
without waiting to be asked.
|
||||
|
||||
---
|
||||
|
||||
## When to Generate a Visual
|
||||
|
||||
Generate a visual whenever the response includes structured data that communicates better
|
||||
visually than as prose. Triggers:
|
||||
|
||||
| Situation | Visual type |
|
||||
|---|---|
|
||||
| Comparing options (cars, tools, services) | Comparison table, bar chart |
|
||||
| Showing trends over time | Line chart |
|
||||
| Displaying locations or dealers | Map with markers |
|
||||
| Scoring or ranking items | Bar chart, sorted table |
|
||||
| Budget breakdowns or proportions | Pie/doughnut chart |
|
||||
| Dashboards summarizing research | React dashboard with stat cards |
|
||||
| Decision-making with weighted criteria | Decision matrix with sliders |
|
||||
|
||||
**Be proactive.** Don't wait to be asked. If you're returning 4+ items with comparable
|
||||
attributes, a visual table is almost always better than a markdown list.
|
||||
|
||||
---
|
||||
|
||||
## Available Libraries (pre-loaded in the runtime)
|
||||
|
||||
| Library | Version | Global | Capabilities |
|
||||
|---|---|---|---|
|
||||
| Tabler CSS | 1.0.0-beta20 | *(stylesheet)* | cards, tables, badges, buttons, forms, grid, dark theme |
|
||||
| Chart.js | v4 | `Chart` | bar, line, pie, radar, doughnut, scatter, polar area |
|
||||
| Leaflet.js | v1.9 | `L` | maps with markers, popups, tile layers (CARTO dark) |
|
||||
| React | v19 | `React` | interactive components with state |
|
||||
| ReactDOM | v19 | `ReactDOM` | DOM rendering for React components |
|
||||
| Babel | v7 | *(automatic)* | JSX compilation when JSX syntax is detected |
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
Wrap your visual in a fenced code block with the `visual` language tag. The content must be
|
||||
a JSON object:
|
||||
|
||||
````
|
||||
```visual
|
||||
{
|
||||
"title": "Display title shown above the visual",
|
||||
"height": 300,
|
||||
"data": { "items": [] },
|
||||
"code": "<div class='card'>...</div><script>...</script>"
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Required | Default | Max | Description |
|
||||
|---|---|---|---|---|
|
||||
| `title` | yes | — | — | Display header shown above the iframe |
|
||||
| `code` | yes | — | — | Complete HTML, JSX, or plain JS to execute |
|
||||
| `data` | no | `{}` | — | Structured object passed into code as `data` |
|
||||
| `height` | no | 300 | 800 | Iframe height in pixels |
|
||||
|
||||
---
|
||||
|
||||
## Code Formats
|
||||
|
||||
The runtime auto-detects the code format and handles it appropriately:
|
||||
|
||||
### 1. React / JSX
|
||||
**Detection:** code contains `<ComponentName` (uppercase tag) or `React.createElement`
|
||||
|
||||
Babel compiles the code, then the runtime renders it into `#root`. Define an `App`
|
||||
component and it auto-renders:
|
||||
|
||||
```
|
||||
function App() {
|
||||
return <div className="card">Hello</div>;
|
||||
}
|
||||
// App is automatically rendered to #root
|
||||
```
|
||||
|
||||
### 2. HTML
|
||||
**Detection:** code contains `<div`, `<table`, `<canvas`, or similar lowercase HTML tags
|
||||
|
||||
The HTML is injected into `#root`. Any `<script>` tags execute with `data` and `mcp`
|
||||
available in scope.
|
||||
|
||||
```
|
||||
<div id="my-table"></div>
|
||||
<script>
|
||||
document.getElementById('my-table').innerHTML = data.items.map(i => i.name).join(', ');
|
||||
</script>
|
||||
```
|
||||
|
||||
### 3. Plain JS
|
||||
**Detection:** code is pure JavaScript without HTML or JSX tags
|
||||
|
||||
Executed as a function receiving `(data, mcp, root, Chart, L)` as parameters:
|
||||
|
||||
```
|
||||
const canvas = document.createElement('canvas');
|
||||
root.appendChild(canvas);
|
||||
new Chart(canvas, { type: 'bar', data: { labels: data.labels, datasets: [...] } });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Available in Code Scope
|
||||
|
||||
| Variable | Type | Description |
|
||||
|---|---|---|
|
||||
| `data` | Object | The `data` field from your JSON (or `{}` if omitted) |
|
||||
| `mcp` | Object | Bridge to the host: `sendMessage(text)`, `callTool(name, args)`, `resize(height)` |
|
||||
| `root` | Element | The `#root` DOM element — your render target |
|
||||
| `Chart` | Class | Chart.js v4 constructor |
|
||||
| `L` | Object | Leaflet.js v1.9 namespace |
|
||||
|
||||
**`mcp.sendMessage(text)`** — sends a user message back to the AI, enabling click-to-explore.
|
||||
**`mcp.callTool(name, args)`** — calls an MCP tool and returns a Promise.
|
||||
**`mcp.resize(height)`** — resizes the iframe (max 800px).
|
||||
|
||||
---
|
||||
|
||||
## Example Patterns
|
||||
|
||||
### 1. Comparison Table (HTML + Tabler CSS)
|
||||
|
||||
````
|
||||
```visual
|
||||
{
|
||||
"title": "Hybrid Cars Under $10K",
|
||||
"height": 280,
|
||||
"data": {
|
||||
"items": [
|
||||
{ "model": "Toyota Prius", "year": 2015, "price": "$8,500", "miles": "72,000", "mpg": 50, "source": "https://cargurus.com" },
|
||||
{ "model": "Honda Civic Hybrid", "year": 2014, "price": "$7,200", "miles": "89,000", "mpg": 44, "source": "https://autotrader.com" },
|
||||
{ "model": "Toyota Camry Hybrid", "year": 2013, "price": "$9,400", "miles": "95,000", "mpg": 41, "source": "https://cars.com" },
|
||||
{ "model": "Ford Fusion Hybrid", "year": 2014, "price": "$8,100", "miles": "81,000", "mpg": 47, "source": "https://cargurus.com" }
|
||||
]
|
||||
},
|
||||
"code": "<div class='theme-dark' style='background:transparent;padding:8px'><div class='card' style='background:#1a1f2e;border-color:#2d3561'><div class='card-body p-2'><div class='table-responsive'><table class='table table-dark table-hover table-sm mb-0'><thead><tr><th>Model</th><th>Year</th><th style='color:#7af7a2'>Price</th><th>Miles</th><th>MPG</th><th>Source</th></tr></thead><tbody id='rows'></tbody></table></div></div></div></div><script>var tbody=document.getElementById('rows');data.items.forEach(function(car){var tr=document.createElement('tr');tr.innerHTML='<td>'+car.model+'</td><td>'+car.year+'</td><td style=\"color:#7af7a2\">'+car.price+'</td><td>'+car.miles+'</td><td>'+car.mpg+'</td><td><a href=\"'+car.source+'\" style=\"color:#7aa2f7\" target=\"_blank\">link</a></td>';tbody.appendChild(tr);});</script>"
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
### 2. Bar Chart (Plain JS + Chart.js)
|
||||
|
||||
````
|
||||
```visual
|
||||
{
|
||||
"title": "Price Comparison by Model",
|
||||
"height": 380,
|
||||
"data": {
|
||||
"labels": ["Prius 2015", "Civic Hybrid 2014", "Camry Hybrid 2013", "Fusion Hybrid 2014"],
|
||||
"values": [8500, 7200, 9400, 8100]
|
||||
},
|
||||
"code": "var canvas = document.createElement('canvas'); root.appendChild(canvas); new Chart(canvas, { type: 'bar', data: { labels: data.labels, datasets: [{ label: 'Price ($)', data: data.values, backgroundColor: 'rgba(122,162,247,0.7)', borderColor: 'rgba(122,162,247,1)', borderWidth: 1 }] }, options: { responsive: true, plugins: { legend: { labels: { color: '#c0c0d0' } } }, scales: { x: { ticks: { color: '#8888aa' }, grid: { color: '#1e2a4a' } }, y: { ticks: { color: '#8888aa' }, grid: { color: '#1e2a4a' } } }, onClick: function(evt, elements) { if (elements.length > 0) { var label = data.labels[elements[0].index]; mcp.sendMessage('Tell me more about ' + label); } } } });"
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
### 3. Location Map (Plain JS + Leaflet)
|
||||
|
||||
````
|
||||
```visual
|
||||
{
|
||||
"title": "Dealers Near Woodinville, WA",
|
||||
"height": 380,
|
||||
"data": {
|
||||
"center": [47.754, -122.163],
|
||||
"zoom": 11,
|
||||
"markers": [
|
||||
{ "lat": 47.754, "lng": -122.163, "title": "Woodinville Auto Center", "description": "Large hybrid selection", "price": "$7,500–$11,000" },
|
||||
{ "lat": 47.770, "lng": -122.205, "title": "Kirkland Motors", "description": "Certified pre-owned", "price": "$8,000–$12,000" },
|
||||
{ "lat": 47.720, "lng": -122.130, "title": "Redmond Auto Group", "description": "Best for Prius models", "price": "$7,200–$10,500" }
|
||||
]
|
||||
},
|
||||
"code": "root.style.height = '340px'; var map = L.map(root).setView(data.center, data.zoom); L.tileLayer('https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}{r}.png', { attribution: '© CARTO', subdomains: 'abcd', maxZoom: 19 }).addTo(map); var bounds = []; data.markers.forEach(function(m) { var marker = L.marker([m.lat, m.lng]).addTo(map); marker.bindPopup('<b>' + m.title + '</b><br>' + m.description + '<br><span style=\"color:#7af7a2\">' + m.price + '</span>'); marker.on('click', function() { mcp.sendMessage('Tell me about ' + m.title); }); bounds.push([m.lat, m.lng]); }); if (bounds.length > 1) { map.fitBounds(bounds, { padding: [20, 20] }); }"
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
### 4. React Dashboard (JSX)
|
||||
|
||||
````
|
||||
```visual
|
||||
{
|
||||
"title": "Research Summary Dashboard",
|
||||
"height": 450,
|
||||
"data": {
|
||||
"cars": [
|
||||
{ "model": "Toyota Prius 2015", "price": "$8,500", "priceNum": 8500, "miles": 72000, "mpg": 50 },
|
||||
{ "model": "Honda Civic Hybrid 2014", "price": "$7,200", "priceNum": 7200, "miles": 89000, "mpg": 44 },
|
||||
{ "model": "Toyota Camry Hybrid 2013", "price": "$9,400", "priceNum": 9400, "miles": 95000, "mpg": 41 },
|
||||
{ "model": "Ford Fusion Hybrid 2014", "price": "$8,100", "priceNum": 8100, "miles": 81000, "mpg": 47 }
|
||||
]
|
||||
},
|
||||
"code": "function StatCard({ label, value }) { return React.createElement('div', { className: 'col-4' }, React.createElement('div', { className: 'card', style: { background: '#1a1f2e', borderColor: '#2d3561' } }, React.createElement('div', { className: 'card-body text-center p-2' }, React.createElement('div', { style: { fontSize: '1.5rem', fontWeight: 'bold', color: '#7af7a2' } }, value), React.createElement('div', { style: { fontSize: '0.75rem', color: '#8888aa' } }, label)))); } function App() { var [sortKey, setSortKey] = React.useState('price'); var cars = data.cars.slice().sort(function(a, b) { return sortKey === 'price' ? a.priceNum - b.priceNum : a.miles - b.miles; }); var avgPrice = Math.round(data.cars.reduce(function(s, c) { return s + c.priceNum; }, 0) / data.cars.length); var minP = Math.min.apply(null, data.cars.map(function(c) { return c.priceNum; })); var maxP = Math.max.apply(null, data.cars.map(function(c) { return c.priceNum; })); return React.createElement('div', { className: 'theme-dark', style: { background: 'transparent', padding: '8px', color: '#e0e0e0' } }, React.createElement('div', { className: 'row g-2 mb-3' }, React.createElement(StatCard, { label: 'Options Found', value: data.cars.length }), React.createElement(StatCard, { label: 'Avg Price', value: '$' + avgPrice.toLocaleString() }), React.createElement(StatCard, { label: 'Price Range', value: '$' + (minP/1000).toFixed(1) + 'k–$' + (maxP/1000).toFixed(1) + 'k' })), React.createElement('div', { className: 'mb-2' }, React.createElement('span', { style: { color: '#8888aa', marginRight: '8px' } }, 'Sort:'), React.createElement('button', { className: 'btn btn-sm ' + (sortKey === 'price' ? 'btn-primary' : 'btn-secondary'), style: { marginRight: '6px' }, onClick: function() { setSortKey('price'); } }, 'Price'), React.createElement('button', { className: 'btn btn-sm ' + (sortKey === 'miles' ? 'btn-primary' : 'btn-secondary'), onClick: function() { setSortKey('miles'); } }, 'Miles')), React.createElement('table', { className: 'table table-dark table-hover table-sm' }, React.createElement('thead', null, React.createElement('tr', null, React.createElement('th', null, 'Model'), React.createElement('th', null, 'Price'), React.createElement('th', null, 'Miles'), React.createElement('th', null, 'MPG'))), React.createElement('tbody', null, cars.map(function(c, i) { return React.createElement('tr', { key: i }, React.createElement('td', null, c.model), React.createElement('td', { style: { color: '#7af7a2' } }, c.price), React.createElement('td', null, c.miles.toLocaleString()), React.createElement('td', null, c.mpg)); })))); } ReactDOM.createRoot(root).render(React.createElement(App));"
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
### 5. Decision Matrix (HTML + interactive weight sliders)
|
||||
|
||||
Interactive weight sliders recalculate scores in real time; the best option is highlighted green.
|
||||
|
||||
````
|
||||
```visual
|
||||
{
|
||||
"title": "Car Decision Matrix",
|
||||
"height": 400,
|
||||
"data": {
|
||||
"criteria": [
|
||||
{ "name": "Price", "weight": 8 },
|
||||
{ "name": "MPG", "weight": 7 },
|
||||
{ "name": "Reliability", "weight": 9 },
|
||||
{ "name": "Cargo", "weight": 5 },
|
||||
{ "name": "Comfort", "weight": 6 }
|
||||
],
|
||||
"options": [
|
||||
{ "name": "Prius 2015", "scores": [7, 10, 9, 6, 7] },
|
||||
{ "name": "Civic Hybrid 2014", "scores": [9, 8, 8, 5, 6] },
|
||||
{ "name": "Camry Hybrid 2013", "scores": [5, 7, 8, 8, 9] },
|
||||
{ "name": "Fusion Hybrid 2014", "scores": [7, 8, 7, 7, 8] }
|
||||
]
|
||||
},
|
||||
"code": "<style>.matrix-table{width:100%;border-collapse:collapse;font-size:0.85rem}.matrix-table th,.matrix-table td{padding:4px 8px;border:1px solid #2d3561;text-align:center}.matrix-table th{background:#1a1f2e;color:#8888aa}.score-cell{color:#c0c0d0}.best{color:#7af7a2;font-weight:bold}.weight-slider{width:80px;accent-color:#7aa2f7}</style><div style='background:transparent;color:#e0e0e0;padding:8px'><div style='margin-bottom:12px;display:flex;flex-wrap:wrap;gap:8px' id='sliders'></div><table class='matrix-table'><thead id='thead'></thead><tbody id='tbody'></tbody></table></div><script>var weights=data.criteria.map(function(c){return c.weight;});function render(){var scores=data.options.map(function(opt){return opt.scores.reduce(function(s,sc,i){return s+sc*weights[i];},0);});var best=Math.max.apply(null,scores);var thead=document.getElementById('thead');thead.innerHTML='<tr><th>Option</th>'+data.criteria.map(function(c){return'<th>'+c.name+'</th>';}).join('')+'<th>Total</th></tr>';var tbody=document.getElementById('tbody');tbody.innerHTML=data.options.map(function(opt,oi){var total=scores[oi];var isBest=total===best;return'<tr><td>'+(isBest?'<span class=\"best\">'+opt.name+' ★</span>':opt.name)+'</td>'+opt.scores.map(function(s){return'<td class=\"score-cell\">'+s+'</td>';}).join('')+'<td class=\"'+(isBest?'best':'score-cell')+'\">'+total+'</td></tr>';}).join('');}var sliderDiv=document.getElementById('sliders');data.criteria.forEach(function(c,i){var label=document.createElement('label');label.style.cssText='display:flex;flex-direction:column;align-items:center;font-size:0.75rem;color:#8888aa';var inp=document.createElement('input');inp.type='range';inp.min=0;inp.max=10;inp.value=c.weight;inp.className='weight-slider';inp.oninput=function(){weights[i]=+this.value;render();};label.appendChild(inp);label.appendChild(document.createTextNode(c.name));sliderDiv.appendChild(label);});render();</script>"
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
---
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Self-contained** — no API fetches; pass all data through the `data` field or hardcode it in `code`
|
||||
- **Dark theme always** — use Tabler `.theme-dark` class, transparent background, light text (`#e0e0e0`)
|
||||
- **Responsive width** — use percentages and `max-width`, never fixed pixel widths for layout
|
||||
- **Max height 800px** — use `mcp.resize(height)` to auto-size after content renders
|
||||
- **No external fetches** — never use `fetch()`, `XMLHttpRequest`, or dynamic script loading in `code`
|
||||
- **Click-to-explore** — chart bars, table rows, and map markers should call `mcp.sendMessage(...)` so the user can drill in
|
||||
|
||||
---
|
||||
|
||||
## When to Use Visuals vs A2UI
|
||||
|
||||
| Need | Choice |
|
||||
|---|---|
|
||||
| Toggle filters for quick filtering | A2UI (`filters` block) |
|
||||
| Simple form (refine search params) | A2UI (`form` block) |
|
||||
| Static table with fewer than 5 rows | A2UI (`table` block) |
|
||||
| Card grid of items with links | A2UI (`cards` block) |
|
||||
| Interactive charts (bar, line, pie) | `visual` block |
|
||||
| sortable / filterable table with many rows | `visual` block |
|
||||
| Comparisons with color-coded columns | `visual` block |
|
||||
| Maps with markers and click interactions | `visual` block |
|
||||
| Dashboards with multiple stat cards | `visual` block |
|
||||
| Decision matrices with weight sliders | `visual` block |
|
||||
Reference in New Issue
Block a user