Add opt-in playwright_execute tool to the CUA agent and CLI#33
Open
dprevoznik wants to merge 10 commits into
Open
Add opt-in playwright_execute tool to the CUA agent and CLI#33dprevoznik wants to merge 10 commits into
dprevoznik wants to merge 10 commits into
Conversation
Exposes a tool that runs Playwright/TypeScript directly against the browser session (via the Kernel SDK browsers.playwright.execute) for steps that are awkward as raw pointer/keyboard actions. Modeled on the existing computer_use_extra navigation tool: defined in cua-ai, executed through the translator, gated by a `playwright` option, and added to keepToolNames so providers retain it in the payload. Enable with the `--playwright` CLI flag. Returns result/stdout/stderr and appends a fresh screenshot so the screenshot loop stays coherent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Drop misleading "Defaults to 60" from timeout_sec description; the actual default lives in the Kernel SDK, not here. - Expose result/stdout/stderr/error on PlaywrightDetails so library consumers can branch on the structured execution result without re-parsing tool content text. - Guard formatPlaywrightResult against non-JSON-serializable returns (e.g. BigInt, circular refs) so a successful Playwright run never becomes a tool-level error. - Sync package-lock.json to match the cua-cli 0.1.1 bump in a7cdc07. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Locals don't persist across calls but the browser session does. Without this, a model could write code in call N assuming variables from call N-1 are still in scope. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Earlier review feedback dropped "Defaults to 60" out of a worry that the default lived in the SDK and could drift. The kernel.sh docs put both the default (60s) and the cap (300s) on the server, so the description is the authoritative place to surface them — the model can't choose a sensible timeout without that anchor. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Schema description tells the model "max 300" but nothing enforced it. A model that ignored the bound would have hit a confusing SDK-level failure depending on server behavior; this clamp keeps the client honest to the documented contract. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- packages/agent: list playwright option alongside computerUseExtra and add a paragraph explaining the tool's behavior and tested-models scope. - packages/ai: list the new tool-definition factory, schema, constants, and CuaPlaywrightInput type in the API surface index. - packages/cli: document --playwright with a short explainer. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
formatPlaywrightResult's JSON.stringify try/catch guarded against non-serializable values, but execution.result came from the SDK after a JSON round trip through the wire — anything that survived that is already JSON-safe, so the catch arm is unreachable. The executePlaywright timeout chain checked typeof === "number" (dead, the parameter is TS-typed number | undefined) and Number.isFinite (redundant — timeoutSec > 0 already rejects NaN, and Math.min handles Infinity). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Empirical results show CUA-specialized providers (Tzafon, Yutori) do emit playwright_execute calls — earlier docs were overly cautious. Yutori in particular demonstrates the failure-as-content design well: it iterated through two wrong-API attempts (page.querySelector, bare document) before reading the stderr/error blocks and landing on page.evaluate(), which throwing would have prevented. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Firetiger deploy monitoring skipped This PR didn't match the auto-monitor filter configured on your GitHub connection:
Reason: PR is in the To monitor this PR anyway, reply with |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Matches executeBatchTool's shape: the trailing translator.screenshot() lives inside the same try/catch as the underlying work, so any failure in the pipeline produces a single wrapped tool error rather than diverging based on which step failed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 565fe01. Configure here.
- executePlaywright: timeout_sec values below 1s previously truncated to 0 and were forwarded to the SDK, which differs from omitting the field. Floor the truncated value at 1s; anything sub-second falls back to "use server default". - Document PlaywrightDetails fields so library consumers know what each one means without reading the executor source. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
playwright_executetool so the model can run Playwright/TypeScript directly against the live browser session — for steps that are awkward as raw pointer/keyboard actions (precise DOM reads, form fills, data extraction, waiting on selectors). It sits alongside the existing computer-use tools rather than replacing them.Execution runs server-side in the browser VM via the Kernel SDK (
client.browsers.playwright.execute), which exposespage/context/browserand lets the codereturna JSON-serializable value. No CDP wiring or local Playwright is needed.It is modeled directly on the existing
computer_use_extranavigation tool:@onkernel/cua-ai—playwright_executetool name,{ code, timeout_sec? }schema, andcreateCuaPlaywrightToolDefinition().@onkernel/cua-agent—InternalComputerTranslator.executePlaywright(), aplaywrightexecutor intools.ts, and theplaywrightoption threaded throughCuaAgent/CuaAgentHarness. The tool name is added tokeepToolNames()so provider payload hooks don't strip it.@onkernel/cua-cli—--playwrightflag and a TUI tool-call preview.Behavior, per the decisions on this:
--playwright(CLI) orplaywright: true(library).result(when present), plusstdout/stderronly when non-empty, anderroronsuccess: false. A reported failure comes back as tool content (not thrown) so the model can adapt; only a thrown SDK error surfaces as a tool error. Library consumers can also read the structuredresult/stdout/stderr/erroroffPlaywrightDetailswithout re-parsing tool content text.timeout_secfollows the documented server contract (default 60s, max 300s); values are clamped client-side so the model can't violate the cap.Naming note
The model-facing wire name is
playwright_execute(snake_case, consistent withcomputer_use_extra/computer_batch), the CLI flag is--playwright, and the option isplaywright.Model support
The tool is advertised as a generic function tool, so any provider that supports function calling alongside its native computer-use API can call it. The
playwright_executename is added tokeepToolNames()so provider payload hooks that filter unknown tools (tzafon/yutori) won't strip it. Verified e2e against:claude-opus-4-7)tzafon.northstar-cua-fast-1.6)n1.5-latest)OpenAI (
gpt-5.5) and Google (gemini-3-flash-preview) are unit-tested but not yet e2e-verified against a live browser.Docs
packages/agent/README.md,packages/ai/README.md, andpackages/cli/README.mdupdated alongside the code.Test plan
npm run typecheck(workspace) passes@onkernel/cua-agentsuite passes, incl. 3 new tests (tool synthesized when enabled; execution formats result/stdout + appends screenshot; failure surfaces as content without throwing)@onkernel/cua-ai(88) and@onkernel/cua-cli(37) suites passcua --playwright) on three providers:claude-opus-4-7) — happy path returnedresult: {"h1":"Example Domain","title":"Example Domain"}in one turn; details carried the structuredresultobject.tzafon.northstar-cua-fast-1.6) — same one-turn happy path. ConfirmskeepToolNames()correctly preserves the tool through tzafon's payload hook.n1.5-latest) — recovered from aTypeError(page.querySelectoris not a function) and aReferenceError(documentnot defined) by reading the failure-as-contentstderr/errorblocks, then arrived at the correctpage.evaluate(...)pattern. Confirms the failure-as-content design closes the iteration loop.success: falsewith the Playwrightstderr/errorcame back as tool content (not thrown), screenshot still appended, model read it and adapted.🤖 Generated with Claude Code
Note
Medium Risk
New remote code execution path against the live browser session when opt-in is enabled; failures are mostly surfaced as tool content but SDK errors can still throw.
Overview
This PR adds an opt-in Playwright escape hatch alongside existing computer-use tools. When enabled (
playwright: trueonCuaAgent/CuaAgentHarness, orcua --playwright), the model gets aplaywright_executetool that runs Playwright/TypeScript in the live Kernel browser viaclient.browsers.playwright.execute.@onkernel/cua-aidefines the tool contract:CuaPlaywrightSchema(code, optionaltimeout_sec),CUA_PLAYWRIGHT_TOOL_NAME, andcreateCuaPlaywrightToolDefinition().@onkernel/cua-agentwires execution:InternalComputerTranslator.executePlaywright()(timeout clamped to 300s), a playwright executor intools.tsthat returns structuredPlaywrightDetailsand model-facing content (result, stdout/stderr, errors without throwing on SDK-reported failure), and a post-call screenshot.playwright_executeis included inkeepToolNames()so provider payload hooks do not strip it. Extra-tool assembly is generalized from navigation-only towithExtraTools.@onkernel/cua-clipasses--playwrightthrough harness setup and shows truncated code in the TUI tool-call preview. Docs and tests cover synthesis, happy path, and failure-as-content behavior. CLI package version bumps to 0.1.1 in the lockfile.Reviewed by Cursor Bugbot for commit d746e8c. Bugbot is set up for automated code reviews on this repo. Configure here.