From c21e4e7f7b00be939cdc3a00bd699cc8c1382445 Mon Sep 17 00:00:00 2001 From: Ken Date: Wed, 27 May 2026 05:11:42 +0000 Subject: [PATCH] fix: browser integration for AI research agent - Dockerfile: Install Google Chrome stable instead of Playwright npm package (playwright-cli defaults to /opt/google/chrome/chrome path). Removed unused Playwright npm package, kept Playwright Python for page automation. - bundle/bundle.md: Fix orchestrator config override with correct key path (session.orchestrator.config.extended_thinking: false) to properly override foundation's extended_thinking: true default, which was causing empty AI text responses. - bundle/agents/researcher.md: Add strong pre-installed environment guardrail telling agent NOT to install packages. Add explicit bash() tool call examples showing correct playwright-cli syntax. Fix snapshot command (removed non-existent -ic flag), change 'type' to 'fill' for form field syntax. - bundle/context/researcher-instructions.md: Enrich from thin pointer to concrete examples with correct bash() wrapping and actual playwright-cli commands. Add 'do NOT install' guardrail. Validated: Agent now opens Chrome browser during research queries, noVNC panel shows live browser activity, AI generates text responses properly. Generated with Amplifier Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com> --- Dockerfile | 11 ++--- bundle/agents/researcher.md | 23 +++++++--- bundle/bundle.md | 8 ++-- bundle/context/researcher-instructions.md | 54 ++++++++++++++++++----- 4 files changed, 72 insertions(+), 24 deletions(-) diff --git a/Dockerfile b/Dockerfile index e18829a..292e24c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -33,13 +33,14 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \ && apt-get install -y --no-install-recommends nodejs \ && rm -rf /var/lib/apt/lists/* -# ── npm global tools: @playwright/cli + Chromium browser ────────────────────── +# ── npm global tools: @playwright/cli + Chrome ──────────────────────────────── # @playwright/cli — token-efficient browser CLI used by research agents # (playwright-cli -s=research open/snapshot/click/type …) -# playwright — provides `playwright install` to download managed Chromium -# Chromium is installed to /root/.cache/ms-playwright/ (default for root user) -RUN npm install -g @playwright/cli playwright \ - && playwright install chromium +# playwright-cli defaults to Chrome at /opt/google/chrome/chrome. Install it directly. +RUN npm install -g @playwright/cli \ + && curl -fsSL https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb -o /tmp/chrome.deb \ + && apt-get update && apt-get install -y /tmp/chrome.deb \ + && rm -f /tmp/chrome.deb && rm -rf /var/lib/apt/lists/* # ── Python tooling ──────────────────────────────────────────────────────────── RUN pip install uv diff --git a/bundle/agents/researcher.md b/bundle/agents/researcher.md index 2c58680..e064ca3 100644 --- a/bundle/agents/researcher.md +++ b/bundle/agents/researcher.md @@ -97,11 +97,24 @@ extract current data, and synthesize findings into clear, actionable results. Th browser runs persistently — tabs stay open between commands, and the user can watch and take over at any time via the noVNC view. +## IMPORTANT: Pre-installed Environment + +`playwright-cli` is **already installed** in this container. Do NOT install anything. +Do NOT run `npm install`, `pip install`, or any package manager. Just use the commands +below directly via the bash tool. + ## Browser Control (playwright-cli) -You control the browser using `playwright-cli` with a persistent named session -(`-s=research`). All commands operate on the same browser session, preserving tabs -and state across the entire conversation. +You control the browser using `playwright-cli` via the bash tool. All commands use +a persistent named session (`-s=research`), preserving tabs and state across the +entire conversation. + +**How to invoke:** Every playwright-cli command runs through the bash tool: +``` +bash("playwright-cli -s=research open https://example.com") +bash("playwright-cli -s=research snapshot") +bash("playwright-cli -s=research click @e5") +``` ### Commands @@ -109,9 +122,9 @@ and state across the entire conversation. |--------|---------| | Open a URL | `playwright-cli -s=research open https://example.com` | | Take a snapshot | `playwright-cli -s=research snapshot` | -| Compact snapshot | `playwright-cli -s=research snapshot -ic` | | Click an element | `playwright-cli -s=research click @e5` | -| Type text | `playwright-cli -s=research type @e3 "search query"` | +| Type text | `playwright-cli -s=research type "search query"` | +| Fill a field | `playwright-cli -s=research fill @e3 "search query"` | | Press a key | `playwright-cli -s=research press Enter` | | Scroll down | `playwright-cli -s=research scroll down` | | Go back | `playwright-cli -s=research back` | diff --git a/bundle/bundle.md b/bundle/bundle.md index 57a21ea..77f2efc 100644 --- a/bundle/bundle.md +++ b/bundle/bundle.md @@ -10,10 +10,10 @@ providers: config: default_model: claude-sonnet-4-20250514 -orchestrator: - module: loop-streaming - config: - extended_thinking: false +session: + orchestrator: + config: + extended_thinking: false includes: - bundle: git+https://github.com/microsoft/amplifier-foundation@main diff --git a/bundle/context/researcher-instructions.md b/bundle/context/researcher-instructions.md index 8a159be..21007d1 100644 --- a/bundle/context/researcher-instructions.md +++ b/bundle/context/researcher-instructions.md @@ -3,22 +3,56 @@ This session runs inside the **Research Workbench**. You have access to the following capabilities: -## Available Capabilities +## CRITICAL: Pre-installed Tools -**Browser** — Live Chromium browser controlled via `playwright-cli -s=research`. -The browser is persistent across commands and visible to the user via VNC (noVNC -panel). The user can take over at any time. +All tools are **already installed**. Do NOT run npm install, pip install, or any +package manager. Just use the commands directly via the bash tool. -**Artifacts** — Persistent markdown documents written to `artifacts//`. +## Browser Control + +Live Chromium browser controlled via `playwright-cli -s=research` through the +bash tool. The browser is persistent across commands and visible to the user via +the noVNC panel. The user can take over at any time. + +**Quick start — open a site and see what's on it:** +``` +bash("playwright-cli -s=research open https://google.com") +bash("playwright-cli -s=research snapshot") +``` + +**Click an element from the snapshot:** +``` +bash("playwright-cli -s=research click @e5") +bash("playwright-cli -s=research snapshot") +``` + +**Type into a field and submit:** +``` +bash("playwright-cli -s=research fill @e3 \"search query\"") +bash("playwright-cli -s=research press Enter") +bash("playwright-cli -s=research snapshot") +``` + +Always snapshot after navigation to get fresh element refs. + +## Artifacts + +Persistent markdown documents written to `artifacts//`. They appear in the right panel and survive across sessions. -**A2UI** — Inline interactive controls (filter chips, tables, card grids, forms) +## A2UI + +Inline interactive controls (filter chips, tables, card grids, forms) rendered directly in chat via `a2ui` code blocks. -**Inline Visuals** — Rich visual output (charts, maps, comparison tables with color +## Inline Visuals + +Rich visual output (charts, maps, comparison tables with color coding) rendered directly in chat via `visual` code blocks. -## Full Command Reference +## Research Workflow -For complete command syntax, workflow guidance, and best practices, see the -researcher agent system prompt. +1. Understand the request — clarify scope and constraints +2. Plan — identify 2-4 sources to consult +3. Browse and extract — use playwright-cli via bash to navigate sites and extract data +4. Present — use A2UI, visuals, or artifacts to deliver results