fix: browser integration for AI research agent
Deploy Research Workbench / deploy (push) Failing after 10m12s
Deploy Research Workbench / deploy (push) Failing after 10m12s
- Dockerfile: Install Google Chrome stable instead of Playwright npm package (playwright-cli defaults to /opt/google/chrome/chrome path). Removed unused Playwright npm package, kept Playwright Python for page automation. - bundle/bundle.md: Fix orchestrator config override with correct key path (session.orchestrator.config.extended_thinking: false) to properly override foundation's extended_thinking: true default, which was causing empty AI text responses. - bundle/agents/researcher.md: Add strong pre-installed environment guardrail telling agent NOT to install packages. Add explicit bash() tool call examples showing correct playwright-cli syntax. Fix snapshot command (removed non-existent -ic flag), change 'type' to 'fill' for form field syntax. - bundle/context/researcher-instructions.md: Enrich from thin pointer to concrete examples with correct bash() wrapping and actual playwright-cli commands. Add 'do NOT install' guardrail. Validated: Agent now opens Chrome browser during research queries, noVNC panel shows live browser activity, AI generates text responses properly. Generated with Amplifier Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This commit is contained in:
@@ -97,11 +97,24 @@ extract current data, and synthesize findings into clear, actionable results. Th
|
||||
browser runs persistently — tabs stay open between commands, and the user can watch
|
||||
and take over at any time via the noVNC view.
|
||||
|
||||
## IMPORTANT: Pre-installed Environment
|
||||
|
||||
`playwright-cli` is **already installed** in this container. Do NOT install anything.
|
||||
Do NOT run `npm install`, `pip install`, or any package manager. Just use the commands
|
||||
below directly via the bash tool.
|
||||
|
||||
## Browser Control (playwright-cli)
|
||||
|
||||
You control the browser using `playwright-cli` with a persistent named session
|
||||
(`-s=research`). All commands operate on the same browser session, preserving tabs
|
||||
and state across the entire conversation.
|
||||
You control the browser using `playwright-cli` via the bash tool. All commands use
|
||||
a persistent named session (`-s=research`), preserving tabs and state across the
|
||||
entire conversation.
|
||||
|
||||
**How to invoke:** Every playwright-cli command runs through the bash tool:
|
||||
```
|
||||
bash("playwright-cli -s=research open https://example.com")
|
||||
bash("playwright-cli -s=research snapshot")
|
||||
bash("playwright-cli -s=research click @e5")
|
||||
```
|
||||
|
||||
### Commands
|
||||
|
||||
@@ -109,9 +122,9 @@ and state across the entire conversation.
|
||||
|--------|---------|
|
||||
| Open a URL | `playwright-cli -s=research open https://example.com` |
|
||||
| Take a snapshot | `playwright-cli -s=research snapshot` |
|
||||
| Compact snapshot | `playwright-cli -s=research snapshot -ic` |
|
||||
| Click an element | `playwright-cli -s=research click @e5` |
|
||||
| Type text | `playwright-cli -s=research type @e3 "search query"` |
|
||||
| Type text | `playwright-cli -s=research type "search query"` |
|
||||
| Fill a field | `playwright-cli -s=research fill @e3 "search query"` |
|
||||
| Press a key | `playwright-cli -s=research press Enter` |
|
||||
| Scroll down | `playwright-cli -s=research scroll down` |
|
||||
| Go back | `playwright-cli -s=research back` |
|
||||
|
||||
+4
-4
@@ -10,10 +10,10 @@ providers:
|
||||
config:
|
||||
default_model: claude-sonnet-4-20250514
|
||||
|
||||
orchestrator:
|
||||
module: loop-streaming
|
||||
config:
|
||||
extended_thinking: false
|
||||
session:
|
||||
orchestrator:
|
||||
config:
|
||||
extended_thinking: false
|
||||
|
||||
includes:
|
||||
- bundle: git+https://github.com/microsoft/amplifier-foundation@main
|
||||
|
||||
@@ -3,22 +3,56 @@
|
||||
This session runs inside the **Research Workbench**. You have access to the
|
||||
following capabilities:
|
||||
|
||||
## Available Capabilities
|
||||
## CRITICAL: Pre-installed Tools
|
||||
|
||||
**Browser** — Live Chromium browser controlled via `playwright-cli -s=research`.
|
||||
The browser is persistent across commands and visible to the user via VNC (noVNC
|
||||
panel). The user can take over at any time.
|
||||
All tools are **already installed**. Do NOT run npm install, pip install, or any
|
||||
package manager. Just use the commands directly via the bash tool.
|
||||
|
||||
**Artifacts** — Persistent markdown documents written to `artifacts/<session_id>/`.
|
||||
## Browser Control
|
||||
|
||||
Live Chromium browser controlled via `playwright-cli -s=research` through the
|
||||
bash tool. The browser is persistent across commands and visible to the user via
|
||||
the noVNC panel. The user can take over at any time.
|
||||
|
||||
**Quick start — open a site and see what's on it:**
|
||||
```
|
||||
bash("playwright-cli -s=research open https://google.com")
|
||||
bash("playwright-cli -s=research snapshot")
|
||||
```
|
||||
|
||||
**Click an element from the snapshot:**
|
||||
```
|
||||
bash("playwright-cli -s=research click @e5")
|
||||
bash("playwright-cli -s=research snapshot")
|
||||
```
|
||||
|
||||
**Type into a field and submit:**
|
||||
```
|
||||
bash("playwright-cli -s=research fill @e3 \"search query\"")
|
||||
bash("playwright-cli -s=research press Enter")
|
||||
bash("playwright-cli -s=research snapshot")
|
||||
```
|
||||
|
||||
Always snapshot after navigation to get fresh element refs.
|
||||
|
||||
## Artifacts
|
||||
|
||||
Persistent markdown documents written to `artifacts/<session_id>/`.
|
||||
They appear in the right panel and survive across sessions.
|
||||
|
||||
**A2UI** — Inline interactive controls (filter chips, tables, card grids, forms)
|
||||
## A2UI
|
||||
|
||||
Inline interactive controls (filter chips, tables, card grids, forms)
|
||||
rendered directly in chat via `a2ui` code blocks.
|
||||
|
||||
**Inline Visuals** — Rich visual output (charts, maps, comparison tables with color
|
||||
## Inline Visuals
|
||||
|
||||
Rich visual output (charts, maps, comparison tables with color
|
||||
coding) rendered directly in chat via `visual` code blocks.
|
||||
|
||||
## Full Command Reference
|
||||
## Research Workflow
|
||||
|
||||
For complete command syntax, workflow guidance, and best practices, see the
|
||||
researcher agent system prompt.
|
||||
1. Understand the request — clarify scope and constraints
|
||||
2. Plan — identify 2-4 sources to consult
|
||||
3. Browse and extract — use playwright-cli via bash to navigate sites and extract data
|
||||
4. Present — use A2UI, visuals, or artifacts to deliver results
|
||||
|
||||
Reference in New Issue
Block a user