diff --git a/docs/design/prompt-runtime-unification/wp-f-playground-mustache/test-plan.md b/docs/design/prompt-runtime-unification/wp-f-playground-mustache/test-plan.md new file mode 100644 index 0000000000..9da69ce4c5 --- /dev/null +++ b/docs/design/prompt-runtime-unification/wp-f-playground-mustache/test-plan.md @@ -0,0 +1,490 @@ +# Manual QA — Playground Mustache + Native JSON + V2 Input UX + +This branch ships: + +- **Backend** (from #4393): mustache rendering, `{{$...}}` JSONPath, frontend `template_format` widening across the editor / chat-message / schema-control surfaces, `vitest` runner in `@agenta/entity-ui`, mustache as the default for new `*_v0` workflow interfaces. +- **Frontend (this branch)**: native-JSON transport surgery in `buildEvaluatorExecutionInputs`, V2-aligned playground input cards (`PlaygroundInputsBody`) behind a feature flag, referenced/draft/unreferenced visibility rule (`playgroundInputsAtomFamily`), playground `TemplateFormatPicker` component. + +This document gives you the fixtures + prompts + scenarios needed to exercise every improvement end-to-end. + +## 0. Prerequisites + +### Branch + worktree + +You're already on `fe-feat/mustache-support` at `.claude/worktrees/fe-feat-mustache-support/`. Confirm: + +```bash +cd .claude/worktrees/fe-feat-mustache-support +git branch --show-current +# fe-feat/mustache-support +git log --oneline -3 +# Should show: ba5fa4c7b chore(frontend): reconcile post-merge ... +# 5483ec07e Merge ... feat/add-mustache-rendering ... +``` + +### Local stack + +Spin up the standard Agenta dev stack from the worktree: + +```bash +# From the worktree root +docker compose -f hosting/docker-compose.dev.yml up +# OR whatever your usual local-dev start command is +``` + +Then run the web dev server pointed at this worktree: + +```bash +cd web/oss +pnpm dev +# Open http://localhost:3000 +``` + +### Enable the V2 input UX feature flag + +The new `PlaygroundInputsBody` rendering is OFF by default (`useNewPlaygroundInputsBodyAtom = false`). To turn it on for QA, you have three options: + +**Option A — temporary code change (recommended for a QA session):** + +Edit `web/packages/agenta-playground-ui/src/state/featureFlags.ts`: + +```diff +- export const useNewPlaygroundInputsBodyAtom = atom(false) ++ export const useNewPlaygroundInputsBodyAtom = atom(true) +``` + +Revert before pushing. The default-flip is an explicit follow-up commit per the design doc. + +**Option B — runtime flip via React DevTools:** + +Open React DevTools → find any component subscribed to `useNewPlaygroundInputsBodyAtom` → use the Jotai DevTools panel to set it `true`. + +**Option C — runtime flip via console:** + +Skip C if you don't already expose the Jotai store globally. + +--- + +## 1. Upload the testset + +The fixture lives at `docs/design/prompt-runtime-unification/wp-f-playground-mustache/testset.json` (same folder as this doc). + +It contains 3 rows (Vanuatu / Kiribati / Switzerland) with deliberately varied column types: + +| Column | Type | Purpose | +|---|---|---| +| `country` | string | plain `{{name}}` mustache substitution | +| `population_thousands` | number | type-chip → `NUMBER`; native-number transport | +| `is_island_nation` | boolean | type-chip → `BOOLEAN`; Switch widget in text mode | +| `geo` | object (nested 2 levels) | mustache `{{geo.region}}` + `{{geo.coordinates.lat}}` — the headline test | +| `languages` | array of strings | Form view; mustache section iteration `{{#languages}}…{{/languages}}` | +| `metadata` | **string that contains JSON** | gap-04: must STAY a string, not be parsed | +| `messages` | array of role-tagged objects | Chat view; tests `ChatMessageList` integration | +| `correct_answer` | string | optional ground-truth column for the evaluator scenario | +| `notes` | string | **unused by default** — exercises the unreferenced-columns footer | + +### Upload steps + +1. In the Agenta web UI, go to **Testsets**. +2. Click **Create new testset** → **Upload file**. +3. Choose the `testset.json` file from the path above. +4. Optional: rename to something like `Mustache QA`. +5. Click **Upload**. +6. Confirm: row count = 3, columns include `country`, `geo`, `messages`, `metadata`, etc. + +**Sanity check — gap-04 in the testset editor:** + +- Open the testset → open the Vanuatu row in the drawer. +- `metadata` should show as type chip **`STRING`** (NOT `OBJECT`), even though its value looks like JSON. If the chip says `STRING`, gap-04 is alive end-to-end on the testset side. +- `geo` should show as **`OBJECT`** with nested `region` / `subregion` / `coordinates` rows. +- `messages` should show as **`MESSAGES`** (or render with the chat editor). + +--- + +## 2. Create the test app (completion-mode mustache) + +The simplest exerciser is a custom completion app whose prompt uses mustache. Newly created apps now default `template_format: "mustache"` (WP-B3 in `*_v0` interfaces). + +1. Go to **Apps** → **Create new app** → **From template** → pick a completion-mode template (e.g. `completion`). +2. Name it `Mustache QA — completion`. +3. Open the new app in the playground. + +### Verify template_format defaults + +Open the variant config panel (the prompt config side, **not** the generations side). Either: + +- Look at the `parameters.prompt.template_format` field, OR +- Open the drawer (focus a prompt) — the new `template_format` dropdown (shipped by #4393's `PromptSchemaControl`) should show **"Prompt Syntax: Mustache"** selected. + +If the dropdown shows curly/jinja2/fstring instead, you're on an existing app that pre-dates the WP-B3 default flip. Either accept that (it works fine, just not mustache) or create a brand-new app. + +### Attach the testset + +1. In the playground, **Generations** panel → **Load testset** → pick `Mustache QA`. +2. You should see one card per testset row (Vanuatu / Kiribati / Switzerland). + +--- + +## 3. Scenarios + +Each scenario lists: the prompt to paste, what to verify, and how to verify. + +### A — Mustache rendering (from #4393) + +These exercise WP-B3's backend renderer + the `extractTemplateVariables` widening shipped by #4393. + +#### A1 — Plain variable substitution + +**Prompt** (paste into the prompt editor's user/system message): + +``` +The country in scope is {{country}}. +``` + +- Run against the Vanuatu testcase. +- **Expected rendered prompt at the backend:** `"The country in scope is Vanuatu."` +- Verify via: the response trace (look at the rendered prompt input the LLM saw). + +#### A2 — Nested object access (the headline mustache feature) + +**Prompt:** + +``` +{{country}} is in the {{geo.region}} region, specifically {{geo.subregion}}. +``` + +- Run against Vanuatu. +- **Expected:** `"Vanuatu is in the Pacific Islands region, specifically Western Melanesia."` +- This is the test that should have FAILED before this branch (transport stringified `geo`, so the renderer received `"{\"region\":\"...\"}"` and `geo.region` resolved to nothing). It must pass now. + +#### A3 — Deep nested access + +**Prompt:** + +``` +{{country}} sits at coordinates lat={{geo.coordinates.lat}}, lng={{geo.coordinates.lng}}. +``` + +- Run against Vanuatu. +- **Expected:** `"Vanuatu sits at coordinates lat=-15.376, lng=166.959."` + +#### A4 — Whole-object insertion (compact JSON) + +**Prompt:** + +``` +Full geo info: {{geo}}. +``` + +- Run against Vanuatu. +- **Expected rendered:** `Full geo info: {"region":"Pacific Islands","subregion":"Western Melanesia","coordinates":{"lat":-15.376,"lng":166.959}}.` +- Per RFC: whole-object insertion renders as compact JSON. + +#### A5 — JSONPath escape hatch + +**Prompt:** + +``` +Region (via JSONPath): {{$.geo.region}}. +First language: {{$.languages[0]}}. +``` + +- Run against Vanuatu. +- **Expected:** `Region (via JSONPath): Pacific Islands. First language: en.` +- `{{$...}}` is pre-rendered before the mustache pass; values are substituted as inert text. +- Per the RFC, the JSONPath root can be a testcase top-level column (keys are spread into the render context). The editor accepts `{{$.geo.region}}` as well as `{{$.inputs.geo.region}}` — both resolve at runtime. The validator only flags actual typos of envelope slot names (e.g. `$.input.country` → suggests `inputs`). + +#### A6 — Mustache section iteration + +**Prompt:** + +``` +Languages: {{#languages}}{{.}} {{/languages}} +``` + +- Run against Vanuatu. +- **Expected:** `Languages: en bi fr ` (trailing space before the closing). + +#### A7 — gap-04: JSON-shaped string stays a string + +**Prompt:** + +``` +Raw metadata: {{metadata}}. +``` + +- Run against Vanuatu. +- **Expected:** `Raw metadata: {"source":"trace","trace_id":"vu-001","latency_ms":520,"confidence":"high"}.` +- The value is INSERTED AS A STRING — not parsed and re-stringified. If you try `{{metadata.source}}` (next scenario), it should NOT resolve. + +#### A8 — Negative test: dotted access on a string + +**Prompt:** + +``` +Source field: {{metadata.source}}. +``` + +- Run against Vanuatu. +- **Expected:** an unresolved-variable error (or empty substitution) — `metadata` is a string, not an object; mustache's dotted-name traversal can't pierce a string. The runtime should treat this as an unresolved tag and surface a clean error. + +### B — Native JSON transport (from this branch, Step 1) + +Verifies `buildEvaluatorExecutionInputs` passes native types through to evaluators. + +#### B1 — Inspect the evaluator request body + +1. Open the playground for an evaluator-equipped variant (or set up an evaluator chain — see §4). +2. Open browser DevTools → **Network** tab. +3. Trigger a run on Vanuatu. +4. Find the request to the evaluator endpoint (look for `/services/evaluators/` or similar). +5. Inspect the **request body** → `inputs` field. + +**Expected:** + +```json +{ + "inputs": { + "geo": { + "region": "Pacific Islands", + "subregion": "Western Melanesia", + "coordinates": {"lat": -15.376, "lng": 166.959} + }, + "languages": ["en", "bi", "fr"], + "metadata": "{\"source\":\"trace\",...}", + "country": "Vanuatu", + ... + } +} +``` + +**Critical:** `inputs.geo` is a JSON OBJECT (not a string `"{\"region\":\"Pacific Islands\",...}"`). `inputs.languages` is a JSON ARRAY (not a string `"[\"en\",\"bi\",\"fr\"]"`). `inputs.metadata` IS a string (gap-04 invariant — preserves user intent). + +#### B2 — Mustache-aware evaluator + +If you have an LLM-as-a-judge evaluator wired into the chain, set its prompt to: + +``` +Your task: check whether the model's answer mentions the {{geo.region}} region. +Model answer: {{prediction}}. +Expected capital: {{correct_answer}}. +Reply YES or NO. +``` + +Set `template_format: "mustache"` and `correct_answer_key: "correct_answer"` on the evaluator. + +- Run against Vanuatu. +- **Expected:** evaluator receives `inputs.geo` as a native object (verified in B1), then mustache resolves `{{geo.region}}` to `"Pacific Islands"`. Before this branch, `geo` arrived as a string and the dotted access failed. + +### C — V2 input UX (feature-flag on) + +Make sure the feature flag is ON (per §0). Reload the playground. + +#### C1 — Bordered cards per variable + +- Open a generation card on the Vanuatu testcase. +- **Expected:** each variable is a bordered card with a header (variable name + type chip + "View as ▾" dropdown) and a body (the editor). + +#### C2 — Type chip vocabulary + +For Vanuatu, you should see chips like: + +| Variable | Chip | +|---|---| +| `country` | `STRING` | +| `population_thousands` | `NUMBER` | +| `is_island_nation` | `BOOLEAN` | +| `geo` | `OBJECT` | +| `languages` | `ARRAY` | +| `metadata` | `STRING` (gap-04 — even though it looks like JSON) | +| `messages` | `MESSAGES` (chat-detected) | + +#### C3 — View-as dropdown options + +Click "View as ▾" on each variable. Confirm the options scale to its kind: + +| Variable kind | Options offered | Default | +|---|---|---| +| string (`country`, `metadata`) | Text, Markdown, JSON, YAML | Text | +| boolean (`is_island_nation`) | Text, JSON, YAML | Text | +| object (`geo`) | Form, JSON, YAML | Form | +| array (`languages`) | Form, JSON, YAML | Form | +| chat (`messages`) | Chat, JSON, YAML | Chat | + +#### C4 — Form mode (object) + +- `geo`'s default view is Form. +- **Expected:** nested fields render inline with their own type chips — `region STRING`, `subregion STRING`, `coordinates OBJECT` → `lat NUMBER`, `lng NUMBER`. +- The 2-level nesting should indent behind a left rail. + +#### C5 — Chat mode (messages) + +- `messages`'s default view is Chat. +- **Expected:** `ChatMessageList` renders with one message per array entry, each with role + content. + +#### C6 — JSON view round-trip + +- Switch `geo` to **JSON** view. +- **Expected:** code editor with pretty-printed JSON (2-space indent, syntax highlight). +- Edit the JSON (e.g. change `region` to `"Oceania"`) → tab out / blur. +- Re-open the variable in the drawer (or switch back to Form view). +- **Expected:** the edit persisted natively (geo.region is now `"Oceania"`). + +#### C7 — YAML view round-trip + +- Switch `geo` to **YAML** view. +- **Expected:** YAML dump with `region:`, `subregion:`, nested `coordinates:`. +- Edit the YAML → tab out. +- **Expected:** parses on edit, stores native value. + +#### C8 — gap-04 in the playground + +- Open `metadata`'s "View as ▾" → switch to **JSON** view. +- **Expected:** the editor shows the JSON inside the string, pretty-printed (because `valueToDisplay(...)`'s json mode attempts `JSON.parse(value)` for strings). +- Make a small edit (change `confidence` to `"medium"`). +- **Expected:** on blur, the value is STILL stored as a string (not parsed and stored as an object). Switch back to Text view to confirm — the value is the raw string text. Type chip stays `STRING`. + +### D — Visibility rule + +#### D1 — Draft variable (referenced + not on testcase) + +- In the prompt editor, add a reference to a variable that doesn't exist on the testcase, e.g. `{{iso_code}}`. +- **Expected:** a new variable card appears in the generations panel with: + - Name: `iso_code` + - Type chip: ambiguous / inferred from undefined → likely `STRING` chip (default) + - A small `draft` text tag in the header + - Empty body (no value yet) + +#### D2 — Draft persistence + +- Type a value into the draft card (e.g. `"VU"`). +- Click **Run**. +- **Expected:** the testcase now carries `iso_code: "VU"`. Reload the page; the `iso_code` column persists on the Vanuatu row. + +#### D3 — Unreferenced columns footer + +- If your prompt does NOT reference `notes`, you should see a footer below all variable cards: + > **▶ 1 unused testcase column hidden because the prompt does not reference them.** +- Click the footer to expand. +- **Expected:** a collapsed card for `notes` appears below. +- If the count is different (e.g. you removed other references), the count updates live as you edit the prompt. + +#### D4 — Referenced ↔ unreferenced transition + +- Add `{{notes}}` to the prompt. +- **Expected:** the `notes` card moves from the footer (unreferenced) up to the expanded cards (referenced). Footer count drops to 0 (footer disappears). +- Remove `{{notes}}` from the prompt. +- **Expected:** `notes` moves back to the unreferenced footer. + +### E — End-to-end mustache + native JSON + +The full integration test — the user-visible win for this whole branch. + +**Prompt:** + +``` +You are a geography research assistant. + +The country in scope is {{country}}. Its region is {{geo.region}}, subregion is {{geo.subregion}}. +Population is {{population_thousands}} thousand. The country speaks: {{#languages}}{{.}} {{/languages}}. + +Coordinates: lat {{geo.coordinates.lat}}, lng {{geo.coordinates.lng}}. + +Reply with a one-sentence answer to the most recent user question. +``` + +1. Confirm `template_format: "mustache"` on the variant (see §2). +2. Set `messages` (or pass `messages` from testcase) as the chat history. +3. Run against Vanuatu. + +**Expected (rendered prompt seen by the LLM):** + +``` +You are a geography research assistant. + +The country in scope is Vanuatu. Its region is Pacific Islands, subregion is Western Melanesia. +Population is 320 thousand. The country speaks: en bi fr . + +Coordinates: lat -15.376, lng 166.959. + +Reply with a one-sentence answer to the most recent user question. +``` + +(Plus the `messages` array tacked on by chat mode.) + +**The model's response** should answer the most recent user message — for Vanuatu, "What is the capital of Vanuatu and its ISO 3166-1 alpha-2 code?" → expected response: something like "Port Vila (VU)". + +If the rendered prompt has bare `{{country}}` / `{{geo.region}}` placeholders (not substituted), template_format is wrong or transport is broken. +If `geo.region` resolves to an empty string, transport is broken (geo arrived as a stringified `"{...}"`, mustache dotted access can't pierce it). + +--- + +## 4. Optional — Evaluator chain setup (for B2 + advanced transport tests) + +Quick recipe to add an LLM-as-a-judge evaluator on top of the test app: + +1. In the playground, **Add** → **Evaluator** → pick `auto_ai_critique` (or `auto_ai_critique_v0`). +2. Wire its inputs: + - `prediction` ← upstream app output + - `correct_answer_key` setting = `"correct_answer"` (so `inputs.correct_answer` resolves from the testcase) +3. Evaluator prompt template (mustache): + ``` + Question: {{messages}} + Expected answer (region context: {{geo.region}}): {{correct_answer}} + Model answer: {{prediction}} + Score: 1 if the model's answer mentions the expected capital, 0 otherwise. + ``` +4. Run against Vanuatu. + +**Critical verification (B1 redux):** check the evaluator's request body. `inputs.geo` must arrive as native object. If it's a stringified JSON, the transport surgery regressed. + +--- + +## 5. Regression checks (existing apps) + +These ensure we didn't break anything that worked before. + +### R1 — Legacy curly app + +- Open any pre-existing app whose `template_format` is `curly`. +- Confirm the picker (in the drawer) shows **Curly** as selected with a "legacy" hint badge. +- Run the existing test cases. +- **Expected:** no behavior change. Curly's literal-key-first lookup still works. + +### R2 — Legacy fstring app + +- Same as R1 but for `fstring`. + +### R3 — Feature flag OFF + +- Set `useNewPlaygroundInputsBodyAtom` back to `false` (revert the temp edit, or flip via devtools). +- Reload the playground. +- **Expected:** the playground falls back to the old `VariableControlAdapter` rendering (borderless per-variable cells, no type chips, no "View as ▾"). No regression. + +--- + +## 6. What's NOT tested by this plan (deferred follow-ups) + +- **ComparisonLayout** — multi-variant side-by-side view still uses `VariableControlAdapter`. Same swap pattern applies; deferred per design doc. +- **Grouped evaluator layout** — `useGroupedLayout === true` branch (evaluator with extracted field ports under envelope sections) still uses `VariableControlAdapter`. +- **TemplateFormatPicker placement in the playground** — the component is built but not yet placed in a specific OSS prompt-config surface (Open Q2 — needs design-team sign-off). The drawer's picker (from #4393) IS testable. +- **Default-flip of the feature flag** — small follow-up commit after you sign off the new UX. + +--- + +## 7. Quick "did everything pass?" checklist + +- [ ] **A1-A6** mustache scenarios all return correctly rendered prompts at the LLM. +- [ ] **A7** `metadata` string is inserted as-is (not parsed). +- [ ] **A8** `{{metadata.source}}` fails with a clear unresolved-variable error. +- [ ] **B1** Network tab shows `inputs.geo` as a native object (not a string). +- [ ] **B2** Mustache-aware evaluator works against the native-JSON inputs. +- [ ] **C1-C5** Each variable type renders with the right chip + view-as options. +- [ ] **C6-C7** JSON ↔ YAML edit round-trips preserve native types. +- [ ] **C8** gap-04 holds: `metadata` stays a STRING even after JSON-view edits. +- [ ] **D1-D2** Draft variables show + persist correctly. +- [ ] **D3-D4** Unreferenced footer updates live as the prompt changes. +- [ ] **E** End-to-end mustache + native JSON renders the expected prompt text. +- [ ] **R1-R3** Legacy formats + feature-flag-off paths unchanged. + +If all of the above pass, this branch is good to ship. diff --git a/docs/design/prompt-runtime-unification/wp-f-playground-mustache/testset.json b/docs/design/prompt-runtime-unification/wp-f-playground-mustache/testset.json new file mode 100644 index 0000000000..654cb313e0 --- /dev/null +++ b/docs/design/prompt-runtime-unification/wp-f-playground-mustache/testset.json @@ -0,0 +1,83 @@ +[ + { + "country": "Vanuatu", + "population_thousands": 320, + "is_island_nation": true, + "geo": { + "region": "Pacific Islands", + "subregion": "Western Melanesia", + "coordinates": { + "lat": -15.376, + "lng": 166.959 + } + }, + "languages": ["en", "bi", "fr"], + "metadata": "{\"source\":\"trace\",\"trace_id\":\"vu-001\",\"latency_ms\":520,\"confidence\":\"high\"}", + "messages": [ + { + "role": "system", + "content": "You are a geography research assistant with access to a country lookup tool. Cite ISO codes when available." + }, + { + "role": "user", + "content": "What is the capital of Vanuatu and its ISO 3166-1 alpha-2 code?" + }, + { + "role": "assistant", + "content": "Port Vila is the capital of Vanuatu. Its ISO 3166-1 alpha-2 code is VU." + } + ], + "correct_answer": "Port Vila", + "notes": "Vanuatu became independent in 1980. Unused-by-default to exercise the unreferenced-columns footer." + }, + { + "country": "Kiribati", + "population_thousands": 120, + "is_island_nation": true, + "geo": { + "region": "Pacific Islands", + "subregion": "Micronesia", + "coordinates": { + "lat": 1.451, + "lng": 172.971 + } + }, + "languages": ["en", "gil"], + "metadata": "{\"source\":\"trace\",\"trace_id\":\"ki-001\",\"latency_ms\":480,\"confidence\":\"high\"}", + "messages": [ + { + "role": "system", + "content": "You are a geography research assistant with access to a country lookup tool. Cite ISO codes when available." + }, + { + "role": "user", + "content": "What is the capital of Kiribati?" + } + ], + "correct_answer": "South Tarawa", + "notes": "Kiribati straddles the equator and the international date line." + }, + { + "country": "Switzerland", + "population_thousands": 8700, + "is_island_nation": false, + "geo": { + "region": "Europe", + "subregion": "Western Europe", + "coordinates": { + "lat": 46.818, + "lng": 8.227 + } + }, + "languages": ["de", "fr", "it", "rm"], + "metadata": "{\"source\":\"trace\",\"trace_id\":\"ch-001\",\"latency_ms\":410,\"confidence\":\"high\"}", + "messages": [ + { + "role": "user", + "content": "What is the capital of Switzerland?" + } + ], + "correct_answer": "Bern", + "notes": "Switzerland is famous for its political neutrality and four national languages." + } +] diff --git a/web/packages/agenta-entities/src/runnable/index.ts b/web/packages/agenta-entities/src/runnable/index.ts index dc3ddcf0f5..9e40114b0a 100644 --- a/web/packages/agenta-entities/src/runnable/index.ts +++ b/web/packages/agenta-entities/src/runnable/index.ts @@ -128,10 +128,12 @@ export { buildEvaluatorExecutionInputs, validateEvaluatorInputs, // Template variable extraction + extractMustacheSectionOpeners, extractTemplateVariables, extractTemplateVariablesFromJson, extractVariablesFromPrompts, extractVariablesFromConfig, + extractSectionOpenersFromConfig, extractVariablesFromEnhancedPrompts, } from "./utils" export type {ExecuteRunnableOptions} from "./utils" diff --git a/web/packages/agenta-entities/src/runnable/portHelpers.ts b/web/packages/agenta-entities/src/runnable/portHelpers.ts index bf5bfacb9b..64d338d10e 100644 --- a/web/packages/agenta-entities/src/runnable/portHelpers.ts +++ b/web/packages/agenta-entities/src/runnable/portHelpers.ts @@ -142,11 +142,14 @@ export interface GroupedTemplateVariable { /** Display label. Same as `key` under this model. */ name: string /** - * Declared shape. `"object"` when any placeholder references a sub-path - * of this group (signals the UI to render a JSON editor); otherwise - * `"string"`. + * Declared shape. + * - `"object"` when any placeholder references a sub-path of this + * group (`{{geo.region}}` → object with `region` sub-path). + * - `"array"` when the name appears ONLY as a mustache section opener + * (`{{#languages}}{{.}}{{/languages}}` → array, no sub-paths). + * - `"string"` otherwise. */ - type: "string" | "object" + type: "string" | "object" | "array" /** * Known sub-paths beneath the group (populated only when `type === "object"`). * Used to seed a shape-hint default in the JSON editor so users see @@ -191,11 +194,35 @@ function parseTemplateExpression(expr: string): ParsedTemplateExpression { const parseSegments = (segments: string[]): ParsedTemplateExpression => { if (segments.length === 0) return {envelope: "inputs", key: ""} - if (segments.length === 1) return {envelope: segments[0], key: ""} + + const first = segments[0] + const firstIsEnvelope = KNOWN_ENVELOPE_SLOTS.has(first) + + if (segments.length === 1) { + // `$.inputs` / `$.outputs` — envelope-only reference. + if (firstIsEnvelope) return {envelope: first, key: ""} + // `$.profile` — testcase-spread key. Per the RFC, testcase + // top-level columns are spread into the render context, so + // they live implicitly under the `inputs` envelope. Treating + // the first segment as the key under `inputs` keeps port + // discovery consistent with envelope-rooted writes. + return {envelope: "inputs", key: first} + } + + if (firstIsEnvelope) { + return { + envelope: first, + key: segments[1], + subPath: segments.length > 2 ? segments.slice(2).join(".") : undefined, + } + } + // Testcase-spread key with a sub-path: `$.profile.name` → + // `{envelope: "inputs", key: "profile", subPath: "name"}`. The + // testcase spread makes this equivalent to `$.inputs.profile.name`. return { - envelope: segments[0], - key: segments[1], - subPath: segments.length > 2 ? segments.slice(2).join(".") : undefined, + envelope: "inputs", + key: first, + subPath: segments.length > 1 ? segments.slice(1).join(".") : undefined, } } @@ -261,9 +288,33 @@ function parseTemplateExpression(expr: string): ParsedTemplateExpression { * `envelope === "inputs"`; other slots are runtime-resolved (backend * populates them from trace / workflow config / etc.). */ -export function groupTemplateVariables(placeholders: string[]): GroupedTemplateVariable[] { +export function groupTemplateVariables( + placeholders: string[], + options?: { + /** Set of names that appeared as mustache section openers + * (`{{#name}}` / `{{^name}}`) in the source template. Used to + * refine type inference: a name referenced ONLY as a section + * opener (no sub-paths) gets `type: "array"` — the iteration + * intent is the strongest signal we have without parsing the + * block body. Names with sub-paths stay `"object"` regardless. */ + sectionOpeners?: Set + }, +): GroupedTemplateVariable[] { const groups = new Map}>() + // Resolve section opener names through `parseTemplateExpression` and + // key them by envelope-scoped `${envelope}.${key}` ids — same identity + // the `groups` map uses below. Otherwise a section opener written as + // `{{#languages}}` would coerce BOTH `inputs.languages` AND + // `outputs.languages` (if both existed in the same prompt) to `array`. + const sectionOpenerIds = new Set() + if (options?.sectionOpeners) { + for (const opener of options.sectionOpeners) { + const parsed = parseTemplateExpression(opener) + if (parsed.key) sectionOpenerIds.add(`${parsed.envelope}.${parsed.key}`) + } + } + for (const placeholder of placeholders) { // Invalid envelope references (e.g. `$.input.xx.abc` — `input` is not // a known envelope slot) don't get an input control. The prompt @@ -289,11 +340,20 @@ export function groupTemplateVariables(placeholders: string[]): GroupedTemplateV return Array.from(groups.values()).map(({envelope, key, subPaths}) => { const subPathList = Array.from(subPaths) + // Type inference priority: + // 1. Sub-paths present → `"object"` (the strongest signal — the + // template addresses specific fields). + // 2. Section opener AND no sub-paths → `"array"` (iteration intent). + // 3. Otherwise → `"string"`. + const groupId = `${envelope}.${key}` + const isSectionOpener = sectionOpenerIds.has(groupId) + const type: GroupedTemplateVariable["type"] = + subPathList.length > 0 ? "object" : isSectionOpener ? "array" : "string" return { envelope, key, name: key, - type: subPathList.length > 0 ? ("object" as const) : ("string" as const), + type, ...(subPathList.length > 0 ? {subPaths: subPathList} : {}), } }) diff --git a/web/packages/agenta-entities/src/runnable/utils.ts b/web/packages/agenta-entities/src/runnable/utils.ts index 64e9889887..648e9b66b3 100644 --- a/web/packages/agenta-entities/src/runnable/utils.ts +++ b/web/packages/agenta-entities/src/runnable/utils.ts @@ -506,17 +506,63 @@ export function extractTemplateVariables( } // curly, jinja2, and mustache all use {{variableName}} for variable substitution - // Linear scan: find '{{', then find '}}', extract the content between them + // Linear scan: find '{{', then find '}}', extract the content between them. + // + // For MUSTACHE, normalise / skip block-level syntax: + // + // keep (strip prefix → variable name): + // - `{{#name}}` — section opener: `name` IS a variable (the iterable + // / truthiness check), it still needs a value. + // - `{{^name}}` — inverted section opener: same — `name` is a variable. + // - `{{&name}}` — unescaped variable: `name` IS the variable. + // + // skip entirely (not variables — structural / inert tokens): + // - `{{/name}}` — section closer (just a boundary marker). + // - `{{!comment}}` — comment. + // - `{{> partial}}` — partial template inclusion (resolved at render). + // - `{{.}}` — implicit iterator (current item, no base name). + // + // For CURLY / JINJA2, none of those prefix characters are valid in + // identifiers — those formats have no section semantics, no implicit + // iterator, no inline comments / partials inside `{{...}}`. If the user + // wrote `{{#items}}` in a curly prompt it's an authoring error (likely + // mustache syntax pasted in). Skip the extraction so no phantom port + // appears in the playground — the user sees the broken token in the + // editor without the FE silently masking it. + // + // The TokenPlugin highlights all of these via its own regex — this filter + // is for PORT DISCOVERY only. The mustache renderer pairs `#`/`^`/`/` + // structurally at render time. let i = 0 while (i < input.length - 1) { if (input[i] === "{" && input[i + 1] === "{") { const start = i + 2 const end = input.indexOf("}}", start) if (end !== -1) { - const variable = input.slice(start, end).trim() - if (variable && !variables.includes(variable)) { - variables.push(variable) + const raw = input.slice(start, end).trim() + + if (templateFormat === "mustache") { + const isSkippablePrefix = /^[/!>]/.test(raw) + const isImplicitIterator = raw === "." + if (raw && !isSkippablePrefix && !isImplicitIterator) { + // Strip section-opener / unescape prefixes — the base + // name is the actual variable that needs a value. + const variable = /^[#^&]/.test(raw) ? raw.slice(1).trim() : raw + if (variable && !variables.includes(variable)) { + variables.push(variable) + } + } + } else { + // curly / jinja2 — only identifier-shaped tokens count. + // Anything starting with mustache-style markers is an + // authoring error in these formats; skip it so the + // playground doesn't surface a port for the bad token. + const startsWithMustacheMarker = /^[#^&/!>.]/.test(raw) + if (raw && !startsWithMustacheMarker && !variables.includes(raw)) { + variables.push(raw) + } } + i = end + 2 } else { // No closing '}}' found, no more variables possible @@ -530,6 +576,47 @@ export function extractTemplateVariables( return variables } +/** + * Extract names that appear as mustache SECTION OPENERS (`{{#name}}` or + * `{{^name}}`) from a template string. + * + * Distinct from `extractTemplateVariables`, which returns every referenced + * placeholder (section openers, plain variables, dotted access). Callers use + * this set as a hint when grouping variables — a name referenced ONLY as a + * section opener with no sub-paths is best surfaced as an `array` port + * (iteration intent) rather than the default `string` port. + * + * Always returns an empty set for non-mustache formats: curly / jinja2 / + * fstring don't have section semantics, so the hint isn't meaningful there. + */ +export function extractMustacheSectionOpeners( + input: string, + templateFormat: TemplateFormat = "curly", +): Set { + const names = new Set() + if (templateFormat !== "mustache") return names + + let i = 0 + while (i < input.length - 1) { + if (input[i] === "{" && input[i + 1] === "{") { + const start = i + 2 + const end = input.indexOf("}}", start) + if (end === -1) break + const raw = input.slice(start, end).trim() + // Section opener (`#name`) or inverted section opener (`^name`). + // `&name` is variable unescape, not a section — exclude. + if (/^[#^]/.test(raw)) { + const name = raw.slice(1).trim() + if (name) names.add(name) + } + i = end + 2 + } else { + i++ + } + } + return names +} + /** * Extract template variables from a JSON object recursively * @param obj - Object to extract variables from @@ -707,6 +794,60 @@ export function extractVariablesFromConfig( return variables } +/** + * Mirror of `extractVariablesFromConfig` that collects mustache SECTION + * OPENERS (`{{#name}}` / `{{^name}}`) across all prompt-like entries in + * config. Used alongside `extractVariablesFromConfig` to feed the + * `sectionOpeners` hint into `groupTemplateVariables`, so a name referenced + * only via section markers (no sub-paths) surfaces as an `array` port + * instead of the default `string` port. + * + * Always returns an empty set for non-mustache prompts — section semantics + * are mustache-specific. + */ +export function extractSectionOpenersFromConfig( + agConfig: Record | undefined, +): Set { + const openers = new Set() + if (!agConfig) return openers + + for (const value of Object.values(agConfig)) { + if (!value || typeof value !== "object" || Array.isArray(value)) continue + const prompt = value as Record + + const rawTf = (prompt.template_format ?? prompt.templateFormat) as string | undefined + const tf = resolveTemplateFormat(rawTf) ?? "curly" + if (tf !== "mustache") continue + + if (!Array.isArray(prompt.messages)) continue + for (const message of prompt.messages) { + if (!message || typeof message !== "object") continue + const content = (message as Record).content + if (typeof content === "string") { + for (const opener of extractMustacheSectionOpeners(content, tf)) { + openers.add(opener) + } + } else if (Array.isArray(content)) { + for (const part of content) { + if (typeof part === "string") { + for (const opener of extractMustacheSectionOpeners(part, tf)) { + openers.add(opener) + } + } else if (part && typeof part === "object") { + const text = (part as Record).text + if (typeof text === "string") { + for (const opener of extractMustacheSectionOpeners(text, tf)) { + openers.add(opener) + } + } + } + } + } + } + } + return openers +} + /** * Synchronize `input_keys` for prompt configs in a parameters object. * @@ -809,7 +950,10 @@ const TESTCASE_OBJECT_KEYS = new Set(["inputs"]) export function buildEvaluatorExecutionInputs(ctx: EvaluatorInputContext): Record { const {testcaseData, upstreamOutput, settings, inputSchema} = ctx - const prediction = normalizeCompact(upstreamOutput) + // RFC invariant: native JSON stays native until template rendering. + // We pass `upstreamOutput` through as-is (object, array, string, primitive) + // and expose it under both `prediction` and `outputs` keys downstream — + // they are the same value, not a stringified copy and a native copy. const schemaProperties = inputSchema?.properties && typeof inputSchema.properties === "object" @@ -822,13 +966,12 @@ export function buildEvaluatorExecutionInputs(ctx: EvaluatorInputContext): Recor inputSchema: inputSchema!, testcaseData, upstreamOutput, - prediction, settings, }) } // Legacy fallback — no schema available - return buildLegacy({testcaseData, prediction, settings}) + return buildLegacy({testcaseData, upstreamOutput, settings}) } /** @@ -840,10 +983,9 @@ function buildFromSchema(ctx: { inputSchema: Record testcaseData: Record upstreamOutput: unknown - prediction: string settings: Record }): Record { - const {schemaProperties, inputSchema, testcaseData, upstreamOutput, prediction, settings} = ctx + const {schemaProperties, inputSchema, testcaseData, upstreamOutput, settings} = ctx const inputs: Record = {} for (const key of Object.keys(schemaProperties)) { @@ -856,13 +998,15 @@ function buildFromSchema(ctx: { const columnName = keySettingValue.startsWith("testcase.") ? keySettingValue.split(".")[1] : keySettingValue - inputs[key] = normalizeCompact(testcaseData[columnName]) + // RFC invariant: native value passes through, not stringified. + inputs[key] = testcaseData[columnName] continue } - // 2. Known upstream output keys + // 2. Known upstream output keys — both `prediction` and `outputs` + // expose the SAME native upstream value; do not stringify either one. if (UPSTREAM_OUTPUT_KEYS.has(key)) { - inputs[key] = key === "prediction" ? prediction : normalizeCompact(upstreamOutput) + inputs[key] = upstreamOutput continue } @@ -889,9 +1033,9 @@ function buildFromSchema(ctx: { } } - // Ensure upstream output is always present in some form + // Ensure upstream output is always present in some form (native, both keys). if (!("prediction" in inputs) && !("outputs" in inputs)) { - inputs.prediction = prediction + inputs.prediction = upstreamOutput inputs.outputs = upstreamOutput } @@ -904,10 +1048,10 @@ function buildFromSchema(ctx: { */ function buildLegacy(ctx: { testcaseData: Record - prediction: string + upstreamOutput: unknown settings: Record }): Record { - const {testcaseData, prediction, settings} = ctx + const {testcaseData, upstreamOutput, settings} = ctx const correctAnswerKey = settings.correct_answer_key const groundTruthKey = @@ -917,12 +1061,12 @@ function buildLegacy(ctx: { ? correctAnswerKey : undefined - const rawGT = groundTruthKey ? testcaseData[groundTruthKey] : undefined - const ground_truth = normalizeCompact(rawGT) + // RFC invariant: native ground-truth value passes through, not stringified. + const ground_truth = groundTruthKey ? testcaseData[groundTruthKey] : undefined const inputs: Record = { ...testcaseData, - prediction, + prediction: upstreamOutput, } if (groundTruthKey) { @@ -1029,24 +1173,6 @@ export function validateEvaluatorInputs(ctx: EvaluatorInputContext): EvaluatorIn return {valid: true, missingInputs: []} } -/** - * Normalize a value to a compact string representation. - * Mirrors DebugSection's `normalizeCompact` helper. - */ -function normalizeCompact(val: unknown): string { - if (val === undefined || val === null) return "" - const str = typeof val === "string" ? val : JSON.stringify(val) - try { - const parsed = JSON.parse(str) - if (parsed && typeof parsed === "object") { - return JSON.stringify(parsed) - } - return str - } catch { - return str - } -} - /** * Transform trace-prefixed keys in evaluator settings. * Strips `trace.` prefix from setting values (e.g. `"trace.spans.output"` → `"spans.output"`). diff --git a/web/packages/agenta-entities/src/workflow/state/molecule.ts b/web/packages/agenta-entities/src/workflow/state/molecule.ts index 55a56521af..f819fafedf 100644 --- a/web/packages/agenta-entities/src/workflow/state/molecule.ts +++ b/web/packages/agenta-entities/src/workflow/state/molecule.ts @@ -50,6 +50,8 @@ import { } from "../../runnable/portHelpers" import {normalizeWorkflowResponse} from "../../runnable/responseHelpers" import { + extractMustacheSectionOpeners, + extractSectionOpenersFromConfig, extractTemplateVariables, extractVariablesFromConfig, resolveTemplateFormat, @@ -771,15 +773,21 @@ function buildEvaluatorFieldPortsFromTemplate(entity: Workflow | null | undefine const fmt = resolveTemplateFormat(rawFmt) ?? "curly" const placeholders: string[] = [] + const sectionOpeners = new Set() for (const message of messages) { const content = message?.content if (typeof content !== "string") continue for (const v of extractTemplateVariables(content, fmt)) { if (!placeholders.includes(v)) placeholders.push(v) } + // Collect mustache `{{#name}}` / `{{^name}}` openers so iteration + // intent surfaces as `type: "array"` for ports with no sub-paths. + for (const opener of extractMustacheSectionOpeners(content, fmt)) { + sectionOpeners.add(opener) + } } - const groups = groupTemplateVariables(placeholders) + const groups = groupTemplateVariables(placeholders, {sectionOpeners}) const ports: RunnablePort[] = [] const seen = new Set() for (const group of groups) { @@ -792,23 +800,36 @@ function buildEvaluatorFieldPortsFromTemplate(entity: Workflow | null | undefine if (RESERVED_FIELD_KEYS.has(group.key)) continue if (seen.has(group.key)) continue seen.add(group.key) + const baseHelpText = `Field referenced in your prompt as \`{{$.inputs.${group.key}}}\`${ + group.type === "string" ? ` or \`{{${group.key}}}\`` : "" + }${ + group.type === "array" + ? ` (used as a mustache section: \`{{#${group.key}}}…{{/${group.key}}}\`)` + : "" + }. Note: this value is merged into the \`inputs\` envelope at runtime, so it also appears inside \`{{inputs}}\` if your prompt renders the whole envelope.` + + // Emit an explicit `array` schema for section-opener ports so the + // empty-shape seed produces `[]` (instead of nothing) and the new + // playground inputs body can show a sensible JSON skeleton on + // drafts. Object ports already get a schema from `subPaths`. + const schema = + group.subPaths && group.subPaths.length > 0 + ? { + type: "object", + properties: Object.fromEntries( + group.subPaths.map((sp) => [sp, {type: "string"}]), + ), + } + : group.type === "array" + ? {type: "array"} + : undefined + ports.push({ key: group.key, name: group.key, type: group.type, - helpText: `Field referenced in your prompt as \`{{$.inputs.${group.key}}}\`${ - group.type === "string" ? ` or \`{{${group.key}}}\`` : "" - }. Note: this value is merged into the \`inputs\` envelope at runtime, so it also appears inside \`{{inputs}}\` if your prompt renders the whole envelope.`, - ...(group.subPaths && group.subPaths.length > 0 - ? { - schema: { - type: "object", - properties: Object.fromEntries( - group.subPaths.map((sp) => [sp, {type: "string"}]), - ), - }, - } - : {}), + helpText: baseHelpText, + ...(schema ? {schema} : {}), }) } return ports @@ -883,14 +904,21 @@ const inputPortsAtomFamily = atomFamily((workflowId: string) => if (params) { const vars = extractVariablesFromConfig(params as Record) if (vars.length > 0) { - return groupTemplateVariables(vars) + const sectionOpeners = extractSectionOpenersFromConfig( + params as Record, + ) + return groupTemplateVariables(vars, {sectionOpeners}) .filter((group) => group.envelope === "inputs") .map((group) => ({ key: group.key, name: group.name, type: group.type, required: true, - ...(group.subPaths ? {schema: buildSubPathSchema(group.subPaths)} : {}), + ...(group.subPaths + ? {schema: buildSubPathSchema(group.subPaths)} + : group.type === "array" + ? {schema: {type: "array"}} + : {}), })) } } @@ -927,14 +955,21 @@ const inputPortsAtomFamily = atomFamily((workflowId: string) => (key) => !systemFields.has(key), ) if (vars.length > 0) { - return groupTemplateVariables(vars) + const sectionOpeners = extractSectionOpenersFromConfig( + params as Record, + ) + return groupTemplateVariables(vars, {sectionOpeners}) .filter((group) => group.envelope === "inputs") .map((group) => ({ key: group.key, name: group.name, type: group.type, required: true, - ...(group.subPaths ? {schema: buildSubPathSchema(group.subPaths)} : {}), + ...(group.subPaths + ? {schema: buildSubPathSchema(group.subPaths)} + : group.type === "array" + ? {schema: {type: "array"}} + : {}), })) } } diff --git a/web/packages/agenta-entities/tests/unit/build-evaluator-execution-inputs.test.ts b/web/packages/agenta-entities/tests/unit/build-evaluator-execution-inputs.test.ts new file mode 100644 index 0000000000..65538263ea --- /dev/null +++ b/web/packages/agenta-entities/tests/unit/build-evaluator-execution-inputs.test.ts @@ -0,0 +1,242 @@ +/** + * Unit tests for buildEvaluatorExecutionInputs — native JSON transport. + * + * Pins the RFC invariant: "native JSON stays native until template rendering." + * Object and array values from testcase / upstream output / ground_truth must + * survive transport as native JSON, NOT as stringified text. This is what + * lets `{{geo.region}}` work in mustache against an object-typed variable. + * + * Pre-Step-1 behavior: `normalizeCompact` stringifies every non-string value. + * Tests in this file FAIL until Step 1 (transport surgery) lands and pass after. + * + * Strings that happen to contain JSON-shaped text MUST stay strings — + * the runtime should never silently parse them. (RFC gap-04.) + */ + +import {describe, it, expect} from "vitest" + +import {buildEvaluatorExecutionInputs, type EvaluatorInputContext} from "../../src/runnable/utils" + +const makeCtx = (overrides: Partial = {}): EvaluatorInputContext => ({ + testcaseData: {}, + upstreamOutput: undefined, + settings: {}, + inputSchema: null, + ...overrides, +}) + +describe("buildEvaluatorExecutionInputs — native JSON transport", () => { + // ── Schema-driven path ────────────────────────────────────────────────── + + describe("with inputSchema (schema-driven)", () => { + it("preserves object value resolved via a *_key setting", () => { + const geo = {region: "Pacific Islands", subregion: "Western Melanesia"} + const ctx = makeCtx({ + testcaseData: {geography_col: geo}, + settings: {geo_key: "geography_col"}, + inputSchema: { + type: "object", + properties: {geo: {type: "object"}}, + }, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.geo).toBe(geo) // same reference — not stringified + expect(typeof inputs.geo).toBe("object") + }) + + it("preserves array value via direct schema property match", () => { + const languages = ["en", "bi", "fr"] + const ctx = makeCtx({ + testcaseData: {languages}, + inputSchema: { + type: "object", + properties: {languages: {type: "array"}}, + }, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.languages).toBe(languages) + expect(Array.isArray(inputs.languages)).toBe(true) + }) + + it("preserves object upstream output as `outputs` / `prediction`", () => { + const upstreamOutput = {answer: "Port Vila", iso: "VU"} + const ctx = makeCtx({ + upstreamOutput, + inputSchema: { + type: "object", + properties: {outputs: {type: "object"}, prediction: {type: "object"}}, + }, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.outputs).toBe(upstreamOutput) + // `prediction` should be the native value too — not a stringified copy. + expect(inputs.prediction).toEqual(upstreamOutput) + expect(typeof inputs.prediction).toBe("object") + }) + + it("keeps strings that look like JSON as strings (gap-04)", () => { + const metadata = '{"source":"trace","trace_id":"vu-001"}' + const ctx = makeCtx({ + testcaseData: {metadata}, + inputSchema: { + type: "object", + properties: {metadata: {type: "string"}}, + }, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.metadata).toBe(metadata) + expect(typeof inputs.metadata).toBe("string") + }) + + it("preserves nested object via additionalProperties spread", () => { + const profile = {name: "Ada", tags: ["admin"]} + const ctx = makeCtx({ + testcaseData: {profile, age: 30}, + inputSchema: { + type: "object", + properties: {age: {type: "number"}}, + // additionalProperties not explicitly false → spread allowed + }, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.profile).toBe(profile) + expect(inputs.age).toBe(30) + }) + + it("preserves primitive types unchanged (number, boolean)", () => { + const ctx = makeCtx({ + testcaseData: {age: 42, is_active: true, ratio: 3.14}, + inputSchema: { + type: "object", + properties: { + age: {type: "number"}, + is_active: {type: "boolean"}, + ratio: {type: "number"}, + }, + }, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.age).toBe(42) + expect(inputs.is_active).toBe(true) + expect(inputs.ratio).toBe(3.14) + }) + }) + + // ── Legacy path (no schema) ────────────────────────────────────────────── + + describe("without inputSchema (legacy fallback)", () => { + it("spreads testcase data preserving native object/array values", () => { + const geo = {region: "Pacific Islands"} + const languages = ["en", "bi"] + const ctx = makeCtx({ + testcaseData: {country: "Vanuatu", geo, languages, age: 320}, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.country).toBe("Vanuatu") + expect(inputs.geo).toBe(geo) + expect(inputs.languages).toBe(languages) + expect(inputs.age).toBe(320) + }) + + it("preserves object upstream output as `prediction`", () => { + const upstreamOutput = {answer: "Port Vila", iso: "VU"} + const ctx = makeCtx({ + testcaseData: {country: "Vanuatu"}, + upstreamOutput, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + // `prediction` is always present in legacy path + expect(inputs.prediction).toEqual(upstreamOutput) + expect(typeof inputs.prediction).toBe("object") + }) + + it("preserves object ground_truth resolved via correct_answer_key", () => { + const correctAnswer = {capital: "Port Vila", iso: "VU"} + const ctx = makeCtx({ + testcaseData: {answer_col: correctAnswer, question: "?"}, + settings: {correct_answer_key: "answer_col"}, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + // Aliased under both `ground_truth` and the original column name. + expect(inputs.ground_truth).toBe(correctAnswer) + expect(inputs.answer_col).toBe(correctAnswer) + expect(typeof inputs.ground_truth).toBe("object") + }) + + it("handles testcase. prefix in correct_answer_key", () => { + const correctAnswer = ["Port Vila"] + const ctx = makeCtx({ + testcaseData: {answer_col: correctAnswer}, + settings: {correct_answer_key: "testcase.answer_col"}, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.ground_truth).toBe(correctAnswer) + expect(Array.isArray(inputs.ground_truth)).toBe(true) + }) + + it("keeps strings that look like JSON as strings (gap-04)", () => { + const metadata = '{"source":"trace"}' + const ctx = makeCtx({ + testcaseData: {metadata, country: "Vanuatu"}, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.metadata).toBe(metadata) + expect(typeof inputs.metadata).toBe("string") + }) + + it("handles null and undefined values without stringifying", () => { + const ctx = makeCtx({ + testcaseData: {nullable: null, missing: undefined, present: "x"}, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.nullable).toBeNull() + expect(inputs.missing).toBeUndefined() + expect(inputs.present).toBe("x") + }) + }) + + // ── Special "inputs" key (TESTCASE_OBJECT_KEYS) ───────────────────────── + + describe('special "inputs" key', () => { + it("passes the whole testcaseData object as `inputs` when schema requests it", () => { + const testcaseData = {country: "Vanuatu", age: 320, geo: {region: "Pacific"}} + const ctx = makeCtx({ + testcaseData, + inputSchema: { + type: "object", + properties: {inputs: {type: "object"}}, + }, + }) + + const inputs = buildEvaluatorExecutionInputs(ctx) + + expect(inputs.inputs).toBe(testcaseData) + // Nested object inside is also native, not stringified. + expect((inputs.inputs as Record).geo).toEqual({region: "Pacific"}) + }) + }) +}) diff --git a/web/packages/agenta-entities/tests/unit/extract-template-variables.test.ts b/web/packages/agenta-entities/tests/unit/extract-template-variables.test.ts new file mode 100644 index 0000000000..160d9a8984 --- /dev/null +++ b/web/packages/agenta-entities/tests/unit/extract-template-variables.test.ts @@ -0,0 +1,184 @@ +/** + * Unit tests for extractTemplateVariables. + * + * Mustache block syntax falls into two buckets for port discovery: + * + * keep (strip prefix → variable name): + * - `{{#name}}` — section opener: `name` IS a variable (the iterable + * truthiness check), it still needs a value. + * - `{{^name}}` — inverted section opener: same — `name` is a variable. + * - `{{&name}}` — unescaped variable: `name` IS the variable. + * + * skip entirely (structural / inert tokens): + * - `{{/name}}` — section closer (boundary marker only). + * - `{{!comment}}` — comment. + * - `{{> partial}}` — partial template inclusion. + * - `{{.}}` — implicit iterator (current item, no base name). + * + * Plain variables, dotted access, and JSONPath are extracted as-is. + */ +import {describe, expect, it} from "vitest" + +import {extractMustacheSectionOpeners, extractTemplateVariables} from "../../src/runnable/utils" + +describe("extractTemplateVariables", () => { + describe("mustache", () => { + it("extracts plain variables", () => { + expect(extractTemplateVariables("Hello {{name}}", "mustache")).toEqual(["name"]) + }) + + it("extracts dotted-name access", () => { + expect(extractTemplateVariables("Region: {{geo.region}}", "mustache")).toEqual([ + "geo.region", + ]) + }) + + it("extracts JSONPath expressions verbatim", () => { + expect(extractTemplateVariables("{{$.geo.region}}", "mustache")).toEqual([ + "$.geo.region", + ]) + }) + + it("extracts section opener `{{#name}}` as a variable", () => { + // The opener IS a variable — `name` needs a value (array to iterate + // or truthy value to render the block). The closer below is just + // a boundary marker so it stays skipped. + expect( + extractTemplateVariables("{{#languages}}{{.}}{{/languages}}", "mustache"), + ).toEqual(["languages"]) + }) + + it("extracts inverted section opener `{{^name}}` as a variable", () => { + expect(extractTemplateVariables("{{^empty}}none{{/empty}}", "mustache")).toEqual([ + "empty", + ]) + }) + + it("extracts unescaped variable `{{&name}}`", () => { + expect(extractTemplateVariables("Raw: {{&html}}", "mustache")).toEqual(["html"]) + }) + + it("skips section close `{{/name}}`", () => { + expect(extractTemplateVariables("hello {{/languages}} world", "mustache")).toEqual([]) + }) + + it("skips comments `{{! ... }}`", () => { + expect(extractTemplateVariables("hello {{! a comment }} world", "mustache")).toEqual([]) + }) + + it("skips partials `{{> name}}`", () => { + // Partials are unsupported at runtime, but the extractor must + // not surface them as variables either. + expect(extractTemplateVariables("hello {{> partial}} world", "mustache")).toEqual([]) + }) + + it("skips the implicit iterator `{{.}}`", () => { + expect(extractTemplateVariables("{{.}}", "mustache")).toEqual([]) + }) + + it("extracts variables AND section openers, skips closers and `.`", () => { + const out = extractTemplateVariables( + "Hi {{name}}, list: {{#items}}- {{.}}{{/items}}. End.", + "mustache", + ) + expect(out).toEqual(["name", "items"]) + }) + + it("deduplicates repeated variables", () => { + expect(extractTemplateVariables("{{a}} {{a}} {{b}}", "mustache")).toEqual(["a", "b"]) + }) + + it("deduplicates section opener against plain reference of same name", () => { + // `{{items}}` followed by `{{#items}}...` should still produce + // a single `items` port. + expect( + extractTemplateVariables( + "Plain: {{items}}; List: {{#items}}- {{.}}{{/items}}.", + "mustache", + ), + ).toEqual(["items"]) + }) + }) + + describe("curly (legacy)", () => { + it("extracts {{name}} variables", () => { + expect(extractTemplateVariables("Hello {{name}}", "curly")).toEqual(["name"]) + }) + + it("extracts dotted access {{user.name}}", () => { + expect(extractTemplateVariables("Hello {{user.name}}", "curly")).toEqual(["user.name"]) + }) + + it("SKIPS mustache-style markers (no section semantics in curly)", () => { + // Curly has no section / inverted-section / comment / partial + // syntax. If the user types `{{#items}}` it's an authoring + // error (mustache syntax pasted in). Don't extract — the + // editor still highlights the bad token visually, but no + // phantom port appears in the playground. The user can fix + // the template instead of being silently "helped". + expect(extractTemplateVariables("{{#items}}{{/items}}", "curly")).toEqual([]) + expect(extractTemplateVariables("{{^empty}}none{{/empty}}", "curly")).toEqual([]) + expect(extractTemplateVariables("{{&unescaped}}", "curly")).toEqual([]) + expect(extractTemplateVariables("{{!comment}}", "curly")).toEqual([]) + expect(extractTemplateVariables("{{> partial}}", "curly")).toEqual([]) + expect(extractTemplateVariables("{{.}}", "curly")).toEqual([]) + }) + + it("still extracts plain variables next to a bad mustache token", () => { + // Mixed authoring: the legit `{{name}}` survives; the + // `{{#items}}` mistake is dropped. + expect(extractTemplateVariables("Hi {{name}}, list: {{#items}}.", "curly")).toEqual([ + "name", + ]) + }) + }) + + describe("fstring", () => { + it("extracts {variable} with single braces", () => { + expect(extractTemplateVariables("Hello {name}", "fstring")).toEqual(["name"]) + }) + + it("treats {{ as escaped literal, not a variable", () => { + expect(extractTemplateVariables("Hello {{name}}", "fstring")).toEqual([]) + }) + }) +}) + +describe("extractMustacheSectionOpeners", () => { + it("returns an empty set for non-mustache formats", () => { + // Section semantics are mustache-specific. Other formats don't get + // the hint even if `#name` appears in their templates. + expect(extractMustacheSectionOpeners("{{#name}}{{/name}}", "curly").size).toBe(0) + expect(extractMustacheSectionOpeners("{{#name}}{{/name}}", "jinja2").size).toBe(0) + expect(extractMustacheSectionOpeners("{#name}{/name}", "fstring").size).toBe(0) + }) + + it("picks up `{{#name}}` openers", () => { + const out = extractMustacheSectionOpeners("{{#languages}}{{.}}{{/languages}}", "mustache") + expect(Array.from(out)).toEqual(["languages"]) + }) + + it("picks up `{{^name}}` inverted-section openers", () => { + const out = extractMustacheSectionOpeners("{{^empty}}none{{/empty}}", "mustache") + expect(Array.from(out)).toEqual(["empty"]) + }) + + it("excludes `{{&name}}` (unescape is a variable, not a section)", () => { + expect(extractMustacheSectionOpeners("{{&html}}", "mustache").size).toBe(0) + }) + + it("excludes closers, comments, partials, the implicit iterator, and plain vars", () => { + expect( + extractMustacheSectionOpeners("{{/name}} {{!c}} {{> p}} {{.}} {{plain}}", "mustache") + .size, + ).toBe(0) + }) + + it("deduplicates repeated openers and mixes plain vars cleanly", () => { + const out = extractMustacheSectionOpeners( + "{{#items}}{{name}}{{/items}} and again {{#items}}{{/items}}", + "mustache", + ) + expect(Array.from(out)).toEqual(["items"]) + }) +}) diff --git a/web/packages/agenta-entities/tests/unit/playground-inputs-visibility.test.ts b/web/packages/agenta-entities/tests/unit/playground-inputs-visibility.test.ts new file mode 100644 index 0000000000..036c5c781e --- /dev/null +++ b/web/packages/agenta-entities/tests/unit/playground-inputs-visibility.test.ts @@ -0,0 +1,166 @@ +/** + * Unit tests for the pure `splitInputsVisibility` rule that powers Step 4 + * of the playground mustache + input UX branch. + * + * Same stopgap-location reasoning as the other Step-2/3 tests: vitest lives + * in agenta-entities; the helper lives in @agenta/playground (the + * package's own test runner is a follow-up). Cross-package relative import + * below is a test-time dep only. + * + * TODO(follow-up): Move alongside the helper once @agenta/playground gets + * its own vitest runner. + */ +import {describe, expect, it} from "vitest" + +import {splitInputsVisibility} from "../../../agenta-playground/src/state/execution/visibility" + +describe("splitInputsVisibility — referenced vs draft vs unreferenced", () => { + describe("referenced + testcase intersection", () => { + it("returns inputs in referencedKeys order with their testcase values", () => { + const result = splitInputsVisibility({ + referencedKeys: ["country", "geo", "messages"], + testcaseData: { + geo: {region: "Pacific"}, + messages: [{role: "user", content: "hi"}], + country: "Vanuatu", + }, + }) + + expect(result.inputs.map((i) => i.name)).toEqual(["country", "geo", "messages"]) + expect(result.inputs[0].value).toBe("Vanuatu") + expect(result.inputs[1].value).toEqual({region: "Pacific"}) + expect(result.inputs[2].value).toEqual([{role: "user", content: "hi"}]) + expect(result.inputs.every((i) => !i.isDraft)).toBe(true) + }) + + it("preserves native object/array values by reference (no clone, no stringify)", () => { + const geo = {region: "Pacific"} + const result = splitInputsVisibility({ + referencedKeys: ["geo"], + testcaseData: {geo}, + }) + expect(result.inputs[0].value).toBe(geo) + }) + }) + + describe("draft variables (referenced but absent from testcase)", () => { + it("annotates referenced names missing from testcase as isDraft: true", () => { + const result = splitInputsVisibility({ + referencedKeys: ["country", "iso_code"], + testcaseData: {country: "Vanuatu"}, + }) + + expect(result.inputs).toEqual([ + {name: "country", value: "Vanuatu"}, + {name: "iso_code", value: undefined, isDraft: true}, + ]) + }) + + it("keeps draft order with referenced order (no reshuffling)", () => { + const result = splitInputsVisibility({ + referencedKeys: ["a", "b", "c"], + testcaseData: {b: 2}, + }) + expect(result.inputs.map((i) => `${i.name}:${i.isDraft ?? false}`)).toEqual([ + "a:true", + "b:false", + "c:true", + ]) + }) + + it("treats a key present with a null value as NOT draft (null is a real value)", () => { + const result = splitInputsVisibility({ + referencedKeys: ["x"], + testcaseData: {x: null}, + }) + expect(result.inputs[0]).toEqual({name: "x", value: null}) + expect(result.inputs[0].isDraft).toBeUndefined() + }) + + it("treats a key present with `undefined` as NOT draft (it's authored)", () => { + // Distinction matters: a testcase row that explicitly carries + // the key (even as undefined) is "authored, just empty". The + // draft state is for keys MISSING from the row entirely. + const result = splitInputsVisibility({ + referencedKeys: ["x"], + testcaseData: {x: undefined}, + }) + expect(result.inputs[0]).toEqual({name: "x", value: undefined}) + expect(result.inputs[0].isDraft).toBeUndefined() + }) + }) + + describe("unreferenced columns", () => { + it("collects testcase columns not in referenced", () => { + const result = splitInputsVisibility({ + referencedKeys: ["country"], + testcaseData: {country: "Vanuatu", population: 320, notes: "n/a"}, + }) + + expect(result.unreferencedColumns).toEqual([ + {name: "population", value: 320}, + {name: "notes", value: "n/a"}, + ]) + }) + + it("preserves native value types in unreferenced (no stringify)", () => { + const profile = {name: "Ada"} + const result = splitInputsVisibility({ + referencedKeys: [], + testcaseData: {profile, tags: ["a", "b"]}, + }) + + expect(result.unreferencedColumns[0].value).toBe(profile) + expect(result.unreferencedColumns[1].value).toEqual(["a", "b"]) + }) + + it("returns empty unreferenced when every testcase key is referenced", () => { + const result = splitInputsVisibility({ + referencedKeys: ["a", "b"], + testcaseData: {a: 1, b: 2}, + }) + expect(result.unreferencedColumns).toEqual([]) + }) + }) + + describe("edge cases", () => { + it("empty referenced + empty testcase → empty result", () => { + expect(splitInputsVisibility({referencedKeys: [], testcaseData: {}})).toEqual({ + inputs: [], + unreferencedColumns: [], + }) + }) + + it("empty referenced + non-empty testcase → all rows go to unreferenced", () => { + const result = splitInputsVisibility({ + referencedKeys: [], + testcaseData: {a: 1, b: 2}, + }) + expect(result.inputs).toEqual([]) + expect(result.unreferencedColumns).toEqual([ + {name: "a", value: 1}, + {name: "b", value: 2}, + ]) + }) + + it("non-empty referenced + empty testcase → all referenced are drafts", () => { + const result = splitInputsVisibility({ + referencedKeys: ["x", "y"], + testcaseData: {}, + }) + expect(result.inputs).toEqual([ + {name: "x", value: undefined, isDraft: true}, + {name: "y", value: undefined, isDraft: true}, + ]) + expect(result.unreferencedColumns).toEqual([]) + }) + + it("does NOT mutate the inputs (referenced list and testcase data are untouched)", () => { + const refs = ["a", "b"] + const data = {a: 1, c: 3} + splitInputsVisibility({referencedKeys: refs, testcaseData: data}) + expect(refs).toEqual(["a", "b"]) + expect(data).toEqual({a: 1, c: 3}) + }) + }) +}) diff --git a/web/packages/agenta-entities/tests/unit/port-helpers.test.ts b/web/packages/agenta-entities/tests/unit/port-helpers.test.ts index a08dda6802..b2c342d2a9 100644 --- a/web/packages/agenta-entities/tests/unit/port-helpers.test.ts +++ b/web/packages/agenta-entities/tests/unit/port-helpers.test.ts @@ -174,10 +174,66 @@ describe("groupTemplateVariables", () => { expect(result).toHaveLength(1) }) - it("ignores invalid template variables", () => { - // '$.invalid.x' is not a known envelope slot, so it should be skipped - const result = groupTemplateVariables(["$.invalid.x"]) - expect(result).toHaveLength(0) + it("ignores invalid template variables (envelope-slot typos)", () => { + // The validator flags envelope-slot typos (e.g. `input` → `inputs`) + // as invalid; groupTemplateVariables skips those. Non-typo roots + // (e.g. `$.geo.region`) are NOW accepted as testcase-spread keys + // per the RFC ("testcase top-level keys are spread into the render + // context") — they DO produce a group rooted at `inputs`. + const typo = groupTemplateVariables(["$.input.x"]) + expect(typo).toHaveLength(0) + }) + + it("treats non-envelope JSONPath roots as testcase-spread inputs", () => { + // RFC canonical: `{{$.profile.name}}` against a `profile` testcase + // column. parseTemplateExpression routes the non-envelope first + // segment under the `inputs` envelope, key = first segment. + const result = groupTemplateVariables(["$.geo.region"]) + expect(result).toHaveLength(1) + expect(result[0].envelope).toBe("inputs") + expect(result[0].key).toBe("geo") + expect(result[0].subPaths).toContain("region") + }) + + describe("sectionOpeners hint (mustache iteration intent)", () => { + it("marks a section-opener-only name as `array`", () => { + // `{{#languages}}{{.}}{{/languages}}` extracts `languages` as a + // plain variable; the sectionOpeners hint tells the grouper it's + // an iteration target → array type. + const result = groupTemplateVariables(["languages"], { + sectionOpeners: new Set(["languages"]), + }) + expect(result).toHaveLength(1) + expect(result[0]).toMatchObject({ + envelope: "inputs", + key: "languages", + type: "array", + }) + }) + + it("keeps sub-pathed names as `object` even when in sectionOpeners", () => { + // `{{#user}}{{user.name}}{{/user}}` — `user` is in sectionOpeners + // BUT also has sub-paths, so it's an object (sub-paths are the + // stronger signal). + const result = groupTemplateVariables(["user", "$.user.name"], { + sectionOpeners: new Set(["user"]), + }) + expect(result).toHaveLength(1) + expect(result[0].type).toBe("object") + expect(result[0].subPaths).toContain("name") + }) + + it("leaves non-opener names as `string`", () => { + const result = groupTemplateVariables(["plain"], { + sectionOpeners: new Set(["other"]), + }) + expect(result[0].type).toBe("string") + }) + + it("is opt-in — no hint behaves exactly as before", () => { + const result = groupTemplateVariables(["languages"]) + expect(result[0].type).toBe("string") + }) }) }) diff --git a/web/packages/agenta-entities/tests/unit/template-variable-validation.test.ts b/web/packages/agenta-entities/tests/unit/template-variable-validation.test.ts new file mode 100644 index 0000000000..543a26a941 --- /dev/null +++ b/web/packages/agenta-entities/tests/unit/template-variable-validation.test.ts @@ -0,0 +1,137 @@ +/** + * Unit tests for validateTemplateVariable. + * + * The validator decides whether a `{{...}}` token gets the "valid" style + * (green) or the "invalid" red-dashed treatment in the Lexical editor. It + * lives in `@agenta/shared/utils/templateVariable.ts`. + * + * Why these tests live in agenta-entities: agenta-shared has no vitest + * runner of its own. Same stopgap pattern as the other tests in this + * folder. We import via the workspace path alias rather than a relative + * path that would couple this test to the package's folder layout. + */ +import {describe, expect, it} from "vitest" + +import {validateTemplateVariable} from "@agenta/shared/utils" + +describe("validateTemplateVariable", () => { + describe("plain names + dot notation", () => { + it("accepts a plain variable name", () => { + expect(validateTemplateVariable("country")).toEqual({valid: true}) + }) + + it("accepts dotted access", () => { + expect(validateTemplateVariable("geo.region")).toEqual({valid: true}) + expect(validateTemplateVariable("geo.coordinates.lat")).toEqual({valid: true}) + }) + + it("rejects an empty expression", () => { + expect(validateTemplateVariable("").valid).toBe(false) + }) + + it("rejects expressions with consecutive separators", () => { + expect(validateTemplateVariable("a..b").valid).toBe(false) + expect(validateTemplateVariable("/a//b").valid).toBe(false) + }) + }) + + describe("JSONPath ($-prefixed)", () => { + it("accepts when rooted at a known envelope slot", () => { + expect(validateTemplateVariable("$.inputs.country").valid).toBe(true) + expect(validateTemplateVariable("$.outputs.answer").valid).toBe(true) + expect(validateTemplateVariable("$.parameters.temperature").valid).toBe(true) + expect(validateTemplateVariable("$.trace.span_id").valid).toBe(true) + }) + + it("accepts when rooted at a testcase top-level column (RFC: keys are spread)", () => { + // Per RFC: testcase top-level keys are spread into the render + // context, so `{{$.profile.name}}` resolves against the spread + // `profile` key. The validator must not gate this. + expect(validateTemplateVariable("$.geo.region").valid).toBe(true) + expect(validateTemplateVariable("$.profile.name").valid).toBe(true) + expect(validateTemplateVariable("$.profile.tags[0]").valid).toBe(true) + expect(validateTemplateVariable("$.country").valid).toBe(true) + }) + + it("accepts the bare root `$` (whole context as compact JSON)", () => { + expect(validateTemplateVariable("$").valid).toBe(true) + }) + + it("rejects `$.` (root with trailing dot, no field)", () => { + // Only the bare `$` is a valid empty form. `$.` has nothing + // after the dot so it can never resolve at render time — + // surface that as an authoring error in the editor. + const result = validateTemplateVariable("$.") + expect(result.valid).toBe(false) + expect(result.reason).toMatch(/has no field/i) + }) + + it("rejects when the root looks like a typo of an envelope slot", () => { + // `input` → `inputs`, `out` → `outputs` etc. These are the + // actionable typo hints the editor surfaces. + const inputTypo = validateTemplateVariable("$.input.country") + expect(inputTypo.valid).toBe(false) + expect(inputTypo.suggestion).toBe("inputs") + + const outTypo = validateTemplateVariable("$.out.answer") + expect(outTypo.valid).toBe(false) + expect(outTypo.suggestion).toBe("outputs") + }) + + it("rejects with a hint that mentions testcase-column escape", () => { + const result = validateTemplateVariable("$.input.country") + expect(result.reason).toMatch(/looks like a typo/i) + // The reason text should mention the testcase-column form so the + // user knows how to express it if `input` is actually their column. + expect(result.reason).toMatch(/testcase column/i) + }) + }) + + describe("JSON Pointer (/-prefixed) — multi-segment", () => { + it("accepts multi-segment paths rooted at a known slot", () => { + expect(validateTemplateVariable("/inputs/country").valid).toBe(true) + expect(validateTemplateVariable("/outputs/answer/iso").valid).toBe(true) + }) + + it("rejects multi-segment paths rooted at an unknown slot", () => { + const result = validateTemplateVariable("/input/country") + expect(result.valid).toBe(false) + expect(result.suggestion).toBe("inputs") + }) + }) + + describe("mustache section close tags (single-segment /-prefixed)", () => { + // {{#languages}}...{{/languages}} — the close tag is a mustache section + // marker, not a JSON Pointer. Before this branch's fix, the validator + // mistook it for a JSON Pointer and rejected it because `languages` + // isn't in KNOWN_ENVELOPE_SLOTS. + + it("accepts `/identifier` (simple mustache section close)", () => { + expect(validateTemplateVariable("/languages").valid).toBe(true) + expect(validateTemplateVariable("/items").valid).toBe(true) + expect(validateTemplateVariable("/users").valid).toBe(true) + }) + + it("accepts `/dotted.identifier` (mustache dotted section close)", () => { + expect(validateTemplateVariable("/profile.name").valid).toBe(true) + expect(validateTemplateVariable("/a.b.c").valid).toBe(true) + }) + + it("accepts `/_underscored`", () => { + expect(validateTemplateVariable("/_private").valid).toBe(true) + }) + + it("does NOT short-circuit multi-segment JSON Pointers", () => { + // Multi-segment paths still go through the envelope-slot check. + const result = validateTemplateVariable("/wrong/path") + expect(result.valid).toBe(false) + expect(result.reason).toMatch(/Unknown envelope slot/i) + }) + + it("rejects identifier-shaped paths that aren't valid identifiers (numeric leading)", () => { + // Numeric-leading isn't a valid mustache identifier or an envelope + // slot — fall through to JSON Pointer validation and reject. + expect(validateTemplateVariable("/123abc").valid).toBe(false) + }) + }) +}) diff --git a/web/packages/agenta-entity-ui/package.json b/web/packages/agenta-entity-ui/package.json index bd7b9d44f4..10e896138d 100644 --- a/web/packages/agenta-entity-ui/package.json +++ b/web/packages/agenta-entity-ui/package.json @@ -20,8 +20,10 @@ "./gatewayTool": "./src/gatewayTool/index.ts", "./modals": "./src/modals/index.ts", "./selection": "./src/selection/index.ts", + "./template-format": "./src/template-format/index.ts", "./testcase": "./src/testcase/index.ts", "./variant": "./src/variant/index.ts", + "./view-types": "./src/view-types/index.ts", "./workflow": "./src/workflow/index.ts" }, "dependencies": { diff --git a/web/packages/agenta-entity-ui/src/DrillInView/components/PlaygroundConfigSection.tsx b/web/packages/agenta-entity-ui/src/DrillInView/components/PlaygroundConfigSection.tsx index 48b800baf2..4add6677aa 100644 --- a/web/packages/agenta-entity-ui/src/DrillInView/components/PlaygroundConfigSection.tsx +++ b/web/packages/agenta-entity-ui/src/DrillInView/components/PlaygroundConfigSection.tsx @@ -40,6 +40,7 @@ import {atom} from "jotai" import {useAtom, useAtomValue, useSetAtom} from "jotai" import yaml from "js-yaml" +import {TemplateFormatPicker, type TemplateFormat} from "../../template-format" import {getModelSchema, getLLMConfigValue, getLLMConfigProperties} from "../SchemaControls" import {feedbackConfigModeAtomFamily} from "../SchemaControls/FeedbackConfigurationControl" import { @@ -1381,6 +1382,26 @@ function PlaygroundConfigSection({ onClick={(e) => e.stopPropagation()} className="flex items-center gap-2 flex-shrink-0" > + {/* Prompt template_format picker — let the user + * switch between Mustache (default), Jinja2, and + * legacy formats (Curly / F-string offered only + * when the prompt is already on one). Writes + * flow through `updatePromptRootField` so the + * value lives at `prompt.template_format` for + * legacy shapes and at the root for canonical. */} + updatePromptRootField("template_format", next)} + /> {!disabled && onRefinePrompt && hasMessages && ( + + ) +} + +const styles = { + trigger: { + display: "inline-flex", + alignItems: "center", + gap: 4, + padding: "0 8px", + height: 24, + borderRadius: 4, + fontSize: 12, + }, + triggerValue: {color: "#051729", fontWeight: 600}, + triggerCaret: {marginTop: 1, opacity: 0.65}, +} + +export default ViewTypeSelect diff --git a/web/packages/agenta-entity-ui/src/view-types/formatters.ts b/web/packages/agenta-entity-ui/src/view-types/formatters.ts new file mode 100644 index 0000000000..c84b74b096 --- /dev/null +++ b/web/packages/agenta-entity-ui/src/view-types/formatters.ts @@ -0,0 +1,172 @@ +/** + * Pure value ↔ display conversions used by surfaces that compose the + * view-mode primitives (e.g. the playground inputs body). + * + * The runtime invariant: native JSON stays native until template rendering + * (RFC). Editors operate on strings; these helpers convert between the two + * per view mode and, on edit-back, preserve the original runtime type when + * possible (so transport sees a number/boolean/null when the user authored one). + * + * No React, no jotai, no antd — just functions over `unknown`. Unit-tested + * in `agenta-entities/tests/unit/playground-inputs-formatters.test.ts` + * (stopgap until entity-ui gets its own test runner). + */ + +import {inferLogicalType, type LogicalType} from "@agenta/shared/utils" +import {dump as yamlDump, load as yamlLoad} from "js-yaml" + +import type {ViewType} from "./viewTypes" + +/** True when the value is an empty array or empty object — the cases for + * which YAML and JSON `[]` / `{}` literals look identical and where seeding + * the YAML buffer doesn't help the user. */ +function isEmptyContainer(value: unknown): boolean { + if (Array.isArray(value)) return value.length === 0 + if (value !== null && typeof value === "object") { + return Object.keys(value as Record).length === 0 + } + return false +} + +/* ── Display: native value → string ──────────────────────────────────── */ + +/** + * Render a value as a string for display in an editor, per view mode. + * + * View modes are pure REPRESENTATION transforms — they NEVER change the + * value's type. A string is a string in every mode; an object is an object + * in every mode. Per the gap-04 invariant ("native JSON stays native"), we + * never auto-parse a JSON-shaped string into an object for display. + * + * - text / markdown: primitives stringify naturally; objects / arrays show + * as compact JSON (matches the runtime's `{{var}}` rendering for + * whole-object insertion). + * - json: `JSON.stringify(value, null, 2)` regardless of type. + * - string "Vanuatu" → `"Vanuatu"` (JSON literal, quoted) + * - string '{"a":1}' → `"{\"a\":1}"` (escaped JSON literal — still a STRING) + * - object {a: 1} → `{\n "a": 1\n}` (multi-line) + * - array ["a", "b"] → `[\n "a",\n "b"\n]` + * - number 42 → `42` + * - boolean true → `true` + * - yaml: `yamlDump(value)` regardless of type. YAML's plain scalars cover + * primitives; objects / arrays produce proper block-style YAML. + * + * Returns `""` for `null` and `undefined` so the editor renders empty. + */ +export function valueToDisplay(value: unknown, mode: ViewType): string { + if (value === undefined || value === null) return "" + + if (mode === "text" || mode === "markdown") { + if (typeof value === "string") return value + if (typeof value === "number" || typeof value === "boolean") return String(value) + try { + return JSON.stringify(value) + } catch { + return String(value) + } + } + + if (mode === "json") { + // No string special-case — strings get JSON-encoded into a literal + // (`"value"` with internal escapes if needed). If the string happens + // to contain JSON-shaped text, we still render it as a STRING + // literal; the type doesn't silently change at display time. + try { + return JSON.stringify(value, null, 2) + } catch { + return String(value) + } + } + + if (mode === "yaml") { + // Empty containers (`[]` / `{}`) only have a flow-style YAML + // representation, which looks identical to JSON literals. Return + // an empty buffer instead so the editor's placeholder guides the + // user to type proper YAML. + if (isEmptyContainer(value)) return "" + // No string special-case here either — yamlDump handles strings as + // plain scalars (`Vanuatu`) and JSON-shaped strings as quoted + // scalars when needed (`'{"a":1}'`). Type preservation, same as + // JSON mode. + try { + return yamlDump(value, {noCompatMode: true, lineWidth: 100}) + } catch { + return String(value) + } + } + + // chat / form view modes are handled by dedicated widgets — this helper + // shouldn't be called for them. Defensive fallback: + try { + return JSON.stringify(value) + } catch { + return String(value) + } +} + +/* ── Edit-back: string → native value, preserving original kind ──────── */ + +/** + * Coerce an edited string back into a native runtime value, preserving the + * original kind when possible. Used by text-mode edits where the editor only + * surfaces strings but the testcase needs to keep its native type. + * + * Rules: + * - originalType `"number"` → `Number(next)` if valid, else the raw string + * (empty string becomes the empty string, + * NOT 0, so the caller can treat it as "clear") + * - originalType `"boolean"` → `true` / `false` for canonical inputs, + * else the raw string (text-mode coercion is + * only relevant for paste edits — the actual + * widget is a Switch) + * - originalType `"null"` → `null` if `next` is empty, else the string + * - everything else → the raw string + * + * For json-object / json-array (which don't get text mode in V2's options), + * the helper still works defensively — but those modes route through + * `parseJsonEdit` or `parseYamlEdit` below, not through `coerceTextEdit`. + */ +export function coerceTextEdit(next: string, originalType: LogicalType): unknown { + if (originalType === "number") { + if (next === "") return "" + const n = Number(next) + return Number.isNaN(n) ? next : n + } + if (originalType === "boolean") { + if (next === "true") return true + if (next === "false") return false + return next + } + if (originalType === "null") { + return next === "" ? null : next + } + return next +} + +/** + * Parse the JSON-mode editor buffer back to a native value. Returns + * `{ok: true, value}` on success, `{ok: false}` on parse failure (caller + * keeps last valid value, mirrors the existing JSON-editor pattern). + */ +export function parseJsonEdit(next: string): {ok: true; value: unknown} | {ok: false} { + try { + return {ok: true, value: JSON.parse(next)} + } catch { + return {ok: false} + } +} + +/** + * Parse the YAML-mode editor buffer back to a native value. + */ +export function parseYamlEdit(next: string): {ok: true; value: unknown} | {ok: false} { + try { + return {ok: true, value: yamlLoad(next)} + } catch { + return {ok: false} + } +} + +/* ── Re-exports for callers that want one import site ─────────────── */ + +export {inferLogicalType, type LogicalType} diff --git a/web/packages/agenta-entity-ui/src/view-types/index.ts b/web/packages/agenta-entity-ui/src/view-types/index.ts new file mode 100644 index 0000000000..571cc833fb --- /dev/null +++ b/web/packages/agenta-entity-ui/src/view-types/index.ts @@ -0,0 +1,39 @@ +/** + * View-types subpath — render-mode vocabulary + components for the playground + * input UX and the testcase drawer. + * + * Import via: + * import {ViewTypeSelect, FormView, getViewOptions} from "@agenta/entity-ui/view-types" + * + * See `viewTypes.ts` for the conceptual model: type chip = inferred kind + * (granular, via `@agenta/shared` + `@agenta/ui` TypeChip); view-as dropdown + * = render mode (this module, 6-way). + */ + +export { + buildEmptyShapeFromSchema, + detectFieldKind, + detectNestedKind, + getDefaultViewForExpectedType, + getDefaultViewForValue, + getViewOptions, + getViewOptionsForExpectedType, + isChatMessagesArray, + type ExpectedType, + type FieldKind, + type NestedKind, + type ViewOption, + type ViewType, +} from "./viewTypes" + +export {ViewTypeSelect} from "./ViewTypeSelect" +export {FormView} from "./FormView" + +export { + coerceTextEdit, + inferLogicalType, + parseJsonEdit, + parseYamlEdit, + valueToDisplay, + type LogicalType, +} from "./formatters" diff --git a/web/packages/agenta-entity-ui/src/view-types/viewTypes.ts b/web/packages/agenta-entity-ui/src/view-types/viewTypes.ts new file mode 100644 index 0000000000..429c2f258a --- /dev/null +++ b/web/packages/agenta-entity-ui/src/view-types/viewTypes.ts @@ -0,0 +1,311 @@ +/** + * Shared view-type vocabulary + per-value option logic for the playground + * input UX (and any other surface that needs a "view as ..." dropdown over + * a typed value). + * + * Promoted from the design-mockups POC (`ProposalV2Views.ts`). See: + * - `docs/design/prompt-runtime-unification/README.md` (WP-F1) + * - Approved design doc in `~/.gstack/projects/Agenta-AI-agenta/...` + * + * The 6 available view types: + * - text | unquoted plain text (string only) + * - markdown | rendered markdown (string only) + * - chat | chat-bubble rendering (messages-shaped arrays only) + * - form | labelled-form rendering (objects) + * - json | structured JSON in a code editor (always available) + * - yaml | structured YAML in a code editor (always available) + * + * `FieldKind` is the 4-way TOP-LEVEL bucketing used to compute the view-mode + * dropdown options for a given value. It is intentionally coarse: + * - string : strings + numbers + nulls (single-primitive values) + * - boolean : true / false + * - object : any structured value (object, non-message array) + * - chat : array of role-tagged message objects + * + * ⚠️ `FieldKind` is NOT the type-chip vocabulary. For the chip, use + * `inferLogicalType` from `@agenta/shared/utils` + `TypeChip` from + * `@agenta/ui`. The chip vocabulary is granular (string/number/boolean/null/ + * json-object/json-array) plus render-hint and state chips. Two distinct + * concerns, two distinct vocabularies. + */ + +/** A user-selectable rendering mode for a value. */ +export type ViewType = "text" | "markdown" | "chat" | "form" | "json" | "yaml" + +/** + * 4-way bucketing used to compute the view-mode dropdown options for a + * top-level value. Internal to the view-options decision. + * + * @see FieldKind doc comment in this file for why this is NOT the chip vocab. + */ +export type FieldKind = "string" | "boolean" | "object" | "chat" + +/** + * Whether the given top-level value should be treated as `chat` (an array of + * role-tagged message objects). Tool-calls and tool-responses still count. + */ +export function isChatMessagesArray(value: unknown): boolean { + if (!Array.isArray(value) || value.length === 0) return false + const VALID_ROLES = new Set(["system", "user", "assistant", "tool", "developer", "function"]) + return value.every((item) => { + if (!item || typeof item !== "object") return false + const role = (item as Record).role + return typeof role === "string" && VALID_ROLES.has(role) + }) +} + +/** + * Reduce the runtime type to the 4-way top-level vocabulary used to decide + * available view modes: + * - chat : array of role-tagged messages + * - object : any structured value (object, plain array) + * - boolean : true / false + * - string : everything else (string, number, null, undefined) + * + * Numbers and nulls are bucketed into `string` because they only appear at + * the top level as single primitives — no separate dropdown options needed. + */ +export function detectFieldKind(value: unknown): FieldKind { + if (isChatMessagesArray(value)) return "chat" + if (typeof value === "boolean") return "boolean" + if (Array.isArray(value)) return "object" + if (value !== null && typeof value === "object") return "object" + return "string" +} + +/** + * Inside a form / nested context we want the precise runtime type so the + * right widget renders (Switch for boolean, InputNumber for number, + * Input.TextArea for string, etc.). + */ +export type NestedKind = "string" | "number" | "boolean" | "null" | "object" | "array" + +export function detectNestedKind(value: unknown): NestedKind { + if (value === null) return "null" + if (typeof value === "string") return "string" + if (typeof value === "number") return "number" + if (typeof value === "boolean") return "boolean" + if (Array.isArray(value)) return "array" + if (typeof value === "object") return "object" + return "string" +} + +/** A dropdown option for the "View as ▾" select. */ +export interface ViewOption { + value: ViewType + label: string + /** Tiny right-aligned hint inside the dropdown row (e.g. "default", "raw"). */ + hint?: string +} + +/** + * Compute the dropdown options for a top-level field. Always includes JSON + + * YAML. Adds Text/Markdown for strings, Chat for chat arrays, Form for objects. + */ +export function getViewOptions(value: unknown): ViewOption[] { + const kind = detectFieldKind(value) + const opts: ViewOption[] = [] + + if (kind === "string") { + opts.push({value: "text", label: "String", hint: "default"}) + opts.push({value: "markdown", label: "Markdown"}) + } else if (kind === "boolean") { + opts.push({value: "text", label: "String", hint: "default"}) + } else if (kind === "chat") { + opts.push({value: "chat", label: "Chat", hint: "default"}) + } else if (kind === "object") { + opts.push({value: "form", label: "Form", hint: "default"}) + } + + opts.push({value: "json", label: "JSON"}) + opts.push({value: "yaml", label: "YAML"}) + + return opts +} + +/** Default view for a top-level field. */ +export function getDefaultViewForValue(value: unknown): ViewType { + return getViewOptions(value)[0]?.value ?? "json" +} + +// ─── Expected-type-aware variants ────────────────────────────────────────── +// +// The plain `getViewOptions` / `getDefaultViewForValue` look only at the +// runtime VALUE. For draft variables (referenced by prompt, not authored on +// the testcase yet) the value is `undefined` — they fall through to the +// "string" branch and produce text-input defaults even when the port schema +// declares the variable as object/array (e.g. `geo` referenced via +// `{{geo.region}}` / `{{geo.coordinates.lat}}`). +// +// The expected-type-aware variants below use the port schema's declared type +// as a fallback when the runtime value is empty (`undefined` / `null` / `""`), +// so draft object ports open as Form by default and chat-shaped arrays open +// as Chat, matching what the user has clearly authored against. +// +// `ExpectedType` is intentionally narrow — it mirrors the port `type` field +// surfaced by `inputPortSchemaMap` (`string` / `number` / `integer` / +// `boolean` / `object` / `array`). Unknown types fall back to value-driven +// behaviour. + +/** Declared port type from the runnable schema. */ +export type ExpectedType = + | "string" + | "number" + | "integer" + | "boolean" + | "object" + | "array" + | undefined + +function isValueEmpty(value: unknown): boolean { + return value === undefined || value === null || value === "" +} + +function fieldKindFromExpected(expected: ExpectedType): FieldKind | null { + if (expected === "object" || expected === "array") return "object" + if (expected === "boolean") return "boolean" + if (expected === "string" || expected === "number" || expected === "integer") return "string" + return null +} + +/** + * Same shape as `getViewOptions(value)`, but when `value` is empty, the + * dropdown is built from `expectedType` instead. This is how a draft + * variable known to be an object opens as `Form` rather than `Text`. + * + * Ordering convention (consistent across every expectedType): the + * kind-specific modes go first, JSON and YAML always live at the BOTTOM. + * Strings get `[String, Markdown, JSON, YAML]`, objects/arrays get + * `[Form, JSON, YAML]`, booleans get `[String, JSON, YAML]`. The default + * mode is decoupled from list order — see `getDefaultViewForExpectedType` + * — so array drafts can still default to JSON without yanking JSON out + * of its conventional bottom-of-the-list slot. + */ +export function getViewOptionsForExpectedType( + value: unknown, + expectedType: ExpectedType, +): ViewOption[] { + if (!isValueEmpty(value)) return getViewOptions(value) + const expectedKind = fieldKindFromExpected(expectedType) + if (!expectedKind) return getViewOptions(value) + + const opts: ViewOption[] = [] + if (expectedKind === "string") { + opts.push({value: "text", label: "String", hint: "default"}) + opts.push({value: "markdown", label: "Markdown"}) + } else if (expectedKind === "boolean") { + opts.push({value: "text", label: "String", hint: "default"}) + } else if (expectedKind === "object") { + opts.push({value: "form", label: "Form"}) + } + // JSON / YAML always at the bottom, in this order, for every kind. + opts.push({value: "json", label: "JSON"}) + opts.push({value: "yaml", label: "YAML"}) + return opts +} + +/** + * Default view mode for a typed draft. Independent of list order — + * `getViewOptionsForExpectedType` keeps a consistent layout regardless + * of which mode is the default. + * + * - object → Form (seeded with empty-shape when a schema is known) + * - array → JSON (FormView has no add-item affordance for `[]`, + * JSON's buffer is the better empty-state UX) + * - string/number/integer/boolean → text ("String" label) + */ +export function getDefaultViewForExpectedType( + value: unknown, + expectedType: ExpectedType, +): ViewType { + if (!isValueEmpty(value)) return getDefaultViewForValue(value) + if (expectedType === "object") return "form" + if (expectedType === "array") return "json" + if (expectedType === "boolean") return "text" + if (expectedType === "string" || expectedType === "number" || expectedType === "integer") { + return "text" + } + return getDefaultViewForValue(value) +} + +// ─── Empty-shape seed from JSON schema ───────────────────────────────────── +// +// When a draft variable references sub-paths (`{{geo.region}}`, +// `{{geo.coordinates.lat}}`) but has no value yet, the playground's synthetic +// port schema describes the expected structure two ways: +// +// - `properties` — top-level keys flattened to `{type: "string"}` placeholders +// - `_pathHints` — original sub-paths (`["region", "coordinates.lat", ...]`) +// preserving the nested shape information `properties` lost +// +// `buildEmptyShapeFromSchema` produces an empty-value object matching that +// expected structure so Form view can show the fields and JSON / YAML modes +// can seed their buffers with the right skeleton. Callers use it as a +// render-only hint — until the user actually edits a field, the testcase +// stays untouched. +// +// Returns `null` for primitive schemas (string / number / boolean) — those +// don't have a "shape" worth seeding; the value-driven helpers handle them. + +/** Build a nested empty-value object from path-hints like `["a.b", "a.c"]`. + * Defensive against malformed entries (non-strings are skipped) — the helper + * receives data sourced from `unknown`-typed schema fragments. */ +function buildShapeFromPathHints(hints: unknown[]): Record { + const out: Record = {} + for (const path of hints) { + if (typeof path !== "string") continue + const segments = path.split(/[.[\]/]/).filter(Boolean) + if (segments.length === 0) continue + let cursor: Record = out + for (let i = 0; i < segments.length; i++) { + const seg = segments[i] + const isLast = i === segments.length - 1 + if (isLast) { + if (!(seg in cursor)) cursor[seg] = "" + } else { + const existing = cursor[seg] + if (typeof existing !== "object" || existing === null || Array.isArray(existing)) { + cursor[seg] = {} + } + cursor = cursor[seg] as Record + } + } + } + return out +} + +/** + * Build a render-only empty-value seed matching the schema's expected + * structure. Returns `null` when there's nothing useful to seed (primitive + * type / missing properties / non-object input). + * + * Order of preference for object schemas: + * 1. `_pathHints` (preserves nested sub-paths) + * 2. `properties` (recursive, flat per level) + */ +export function buildEmptyShapeFromSchema(schema: unknown): unknown { + if (!schema || typeof schema !== "object") return null + const s = schema as { + type?: string + properties?: Record + items?: unknown + _pathHints?: unknown + } + + if (s.type === "object" && Array.isArray(s._pathHints) && s._pathHints.length > 0) { + return buildShapeFromPathHints(s._pathHints) + } + + if (s.type === "object" && s.properties) { + const out: Record = {} + for (const [key, prop] of Object.entries(s.properties)) { + const nested = buildEmptyShapeFromSchema(prop) + out[key] = nested ?? "" + } + return out + } + + if (s.type === "array") return [] + // Primitive / unknown schemas — no seed worth emitting. + return null +} diff --git a/web/packages/agenta-entity-ui/tests/unit/playground-inputs-formatters.test.ts b/web/packages/agenta-entity-ui/tests/unit/playground-inputs-formatters.test.ts new file mode 100644 index 0000000000..ced27e4879 --- /dev/null +++ b/web/packages/agenta-entity-ui/tests/unit/playground-inputs-formatters.test.ts @@ -0,0 +1,197 @@ +/** + * Unit tests for the pure formatters in @agenta/entity-ui/view-types. + * + * Runs under @agenta/entity-ui's own vitest runner (added by #4393's + * vitest.config.ts). + */ +import {describe, expect, it} from "vitest" + +import { + coerceTextEdit, + parseJsonEdit, + parseYamlEdit, + valueToDisplay, +} from "../../src/view-types/formatters" + +describe("formatters: valueToDisplay", () => { + describe("nullish handling", () => { + it("returns '' for null and undefined in every mode", () => { + for (const mode of ["text", "markdown", "json", "yaml"] as const) { + expect(valueToDisplay(null, mode)).toBe("") + expect(valueToDisplay(undefined, mode)).toBe("") + } + }) + }) + + describe("text and markdown mode", () => { + it("passes strings through unchanged", () => { + expect(valueToDisplay("hello", "text")).toBe("hello") + expect(valueToDisplay("hello", "markdown")).toBe("hello") + }) + + it("converts primitives via String()", () => { + expect(valueToDisplay(42, "text")).toBe("42") + expect(valueToDisplay(true, "text")).toBe("true") + expect(valueToDisplay(false, "text")).toBe("false") + }) + + it("renders objects/arrays as compact JSON (matches backend whole-value insertion)", () => { + expect(valueToDisplay({a: 1}, "text")).toBe('{"a":1}') + expect(valueToDisplay([1, 2], "text")).toBe("[1,2]") + }) + + it("keeps a JSON-shaped string as the raw string (gap-04: strings stay strings)", () => { + const raw = '{"x":1}' + expect(valueToDisplay(raw, "text")).toBe(raw) + }) + }) + + describe("json mode", () => { + it("pretty-prints native objects and arrays", () => { + expect(valueToDisplay({a: 1}, "json")).toBe('{\n "a": 1\n}') + expect(valueToDisplay([1, 2], "json")).toBe("[\n 1,\n 2\n]") + }) + + it("renders strings as JSON-literal (gap-04: strings stay strings — never parse)", () => { + // Plain string → quoted JSON literal. Crucially NOT the raw + // unquoted text, which the JSON code editor flags as a syntax + // error. + expect(valueToDisplay("Vanuatu", "json")).toBe('"Vanuatu"') + }) + + it("renders JSON-shaped strings AS STRINGS (never auto-parse into objects)", () => { + // The metadata bug Mahmoud flagged: stringified JSON must NOT be + // mistaken for an object. The display preserves the string type + // by JSON-encoding the string literal (outer quotes, escaped + // inner quotes) instead of parsing + pretty-printing. + expect(valueToDisplay('{"a":1}', "json")).toBe('"{\\"a\\":1}"') + }) + + it("stringifies primitives as JSON literals", () => { + expect(valueToDisplay(42, "json")).toBe("42") + expect(valueToDisplay(true, "json")).toBe("true") + expect(valueToDisplay(false, "json")).toBe("false") + }) + }) + + describe("yaml mode", () => { + it("dumps native objects as YAML", () => { + const out = valueToDisplay({a: 1, b: "two"}, "yaml") + expect(out).toContain("a: 1") + expect(out).toContain("b: two") + }) + + it("dumps strings AS STRINGS (never auto-parse JSON-shaped strings)", () => { + // gap-04: type preservation in display. A string containing + // JSON-shaped text gets yamlDump'd as a YAML scalar (quoted + // because the leading `{` would otherwise be ambiguous), NOT + // converted to a YAML mapping. + const out = valueToDisplay('{"a":1}', "yaml") + // The result is a YAML scalar — quoted or escaped depending on + // js-yaml's choice, but it must NOT produce a `a: 1` mapping. + expect(out).not.toMatch(/^a:\s/m) + // And the original string content survives a YAML re-parse. + // (Sanity check that we're dumping the string, not anything else.) + expect(out).toContain('{"a":1}') + }) + + it("dumps plain strings as YAML plain scalars", () => { + expect(valueToDisplay("hello world", "yaml").trim()).toBe("hello world") + }) + + it("returns empty string for empty containers (no flow-style `[]` / `{}` literal)", () => { + // `js-yaml.dump([])` and `dump({})` only produce flow-style + // literals because block style requires at least one item. The + // result looks identical to JSON — confusing the user when they + // explicitly picked YAML mode. Render as empty so the editor's + // placeholder takes over and the user types fresh YAML. + expect(valueToDisplay([], "yaml")).toBe("") + expect(valueToDisplay({}, "yaml")).toBe("") + }) + + it("preserves string-typed empty-container LITERALS (gap-04)", () => { + // `"[]"` is a STRING — it should NOT be parsed and then + // emptied. It survives as a YAML string scalar. + const out = valueToDisplay("[]", "yaml") + expect(out).not.toBe("") // not suppressed (it's a string, not an empty array) + expect(out).toContain("[]") + }) + + it("still dumps non-empty arrays / objects as block-style YAML", () => { + const arr = valueToDisplay(["en", "fr"], "yaml") + expect(arr).toContain("- en") + expect(arr).toContain("- fr") + const obj = valueToDisplay({foo: "bar"}, "yaml") + expect(obj).toContain("foo: bar") + }) + }) +}) + +describe("formatters: coerceTextEdit", () => { + it("preserves number type for numeric originals", () => { + expect(coerceTextEdit("320", "number")).toBe(320) + expect(coerceTextEdit("3.14", "number")).toBe(3.14) + expect(coerceTextEdit("0", "number")).toBe(0) + expect(coerceTextEdit("-5", "number")).toBe(-5) + }) + + it("falls back to the raw string for invalid number edits", () => { + expect(coerceTextEdit("not-a-number", "number")).toBe("not-a-number") + }) + + it("returns '' (empty string sentinel) for empty number edits", () => { + // Number(\"\") is 0, but that's a worse default than letting the + // caller decide. Return empty string so caller can treat as "clear". + expect(coerceTextEdit("", "number")).toBe("") + }) + + it("coerces 'true'/'false' for boolean originals", () => { + expect(coerceTextEdit("true", "boolean")).toBe(true) + expect(coerceTextEdit("false", "boolean")).toBe(false) + }) + + it("keeps the raw string for non-canonical boolean edits", () => { + expect(coerceTextEdit("yes", "boolean")).toBe("yes") + expect(coerceTextEdit("1", "boolean")).toBe("1") + }) + + it("coerces empty back to null for null originals", () => { + expect(coerceTextEdit("", "null")).toBeNull() + expect(coerceTextEdit("something", "null")).toBe("something") + }) + + it("passes strings through unchanged", () => { + expect(coerceTextEdit("hello", "string")).toBe("hello") + expect(coerceTextEdit('{"x":1}', "string")).toBe('{"x":1}') + }) +}) + +describe("formatters: parseJsonEdit", () => { + it("returns {ok: true, value} for valid JSON", () => { + expect(parseJsonEdit('{"a":1}')).toEqual({ok: true, value: {a: 1}}) + expect(parseJsonEdit("[1,2,3]")).toEqual({ok: true, value: [1, 2, 3]}) + expect(parseJsonEdit("42")).toEqual({ok: true, value: 42}) + expect(parseJsonEdit("true")).toEqual({ok: true, value: true}) + expect(parseJsonEdit("null")).toEqual({ok: true, value: null}) + }) + + it("returns {ok: false} on parse failure", () => { + expect(parseJsonEdit("not json")).toEqual({ok: false}) + expect(parseJsonEdit("{a:1}")).toEqual({ok: false}) + expect(parseJsonEdit("")).toEqual({ok: false}) + }) +}) + +describe("formatters: parseYamlEdit", () => { + it("returns {ok: true, value} for valid YAML", () => { + const res = parseYamlEdit("a: 1\nb: two") + expect(res.ok).toBe(true) + if (res.ok) expect(res.value).toEqual({a: 1, b: "two"}) + }) + + it("accepts plain strings (YAML treats them as scalars)", () => { + const res = parseYamlEdit("hello world") + expect(res.ok).toBe(true) + if (res.ok) expect(res.value).toBe("hello world") + }) +}) diff --git a/web/packages/agenta-entity-ui/tests/unit/view-types.test.ts b/web/packages/agenta-entity-ui/tests/unit/view-types.test.ts new file mode 100644 index 0000000000..3767c2e465 --- /dev/null +++ b/web/packages/agenta-entity-ui/tests/unit/view-types.test.ts @@ -0,0 +1,353 @@ +/** + * Unit tests for the view-types primitives in @agenta/entity-ui/view-types. + * + * Runs under @agenta/entity-ui's own vitest runner (added by #4393's + * vitest.config.ts). Previously these tests lived as a stopgap in + * @agenta/entities/tests/unit/ because entity-ui had no runner — that + * limitation was lifted by #4393, so the tests moved to their natural home. + */ +import {describe, expect, it} from "vitest" + +import { + buildEmptyShapeFromSchema, + detectFieldKind, + detectNestedKind, + getDefaultViewForExpectedType, + getDefaultViewForValue, + getViewOptions, + getViewOptionsForExpectedType, + isChatMessagesArray, +} from "../../src/view-types/viewTypes" + +describe("view-types: isChatMessagesArray", () => { + it("detects a basic role-tagged messages array", () => { + const messages = [ + {role: "system", content: "you are a helper"}, + {role: "user", content: "hi"}, + {role: "assistant", content: "hello!"}, + ] + expect(isChatMessagesArray(messages)).toBe(true) + }) + + it("accepts every supported role (system/user/assistant/tool/developer/function)", () => { + const messages = [ + {role: "system", content: "x"}, + {role: "user", content: "x"}, + {role: "assistant", content: "x"}, + {role: "tool", content: "x"}, + {role: "developer", content: "x"}, + {role: "function", content: "x"}, + ] + expect(isChatMessagesArray(messages)).toBe(true) + }) + + it("rejects empty arrays", () => { + expect(isChatMessagesArray([])).toBe(false) + }) + + it("rejects arrays of plain objects without role", () => { + expect(isChatMessagesArray([{content: "x"}, {content: "y"}])).toBe(false) + }) + + it("rejects arrays with an unrecognized role on any item", () => { + expect( + isChatMessagesArray([ + {role: "user", content: "x"}, + {role: "bogus", content: "y"}, + ]), + ).toBe(false) + }) + + it("rejects non-arrays (string, object, null, undefined)", () => { + expect(isChatMessagesArray("hi")).toBe(false) + expect(isChatMessagesArray({role: "user"})).toBe(false) + expect(isChatMessagesArray(null)).toBe(false) + expect(isChatMessagesArray(undefined)).toBe(false) + }) +}) + +describe("view-types: detectFieldKind (top-level 4-way bucketing)", () => { + it("buckets strings, numbers, nulls into 'string'", () => { + expect(detectFieldKind("hello")).toBe("string") + expect(detectFieldKind(42)).toBe("string") + expect(detectFieldKind(null)).toBe("string") + expect(detectFieldKind(undefined)).toBe("string") + }) + + it("returns 'boolean' for booleans", () => { + expect(detectFieldKind(true)).toBe("boolean") + expect(detectFieldKind(false)).toBe("boolean") + }) + + it("returns 'object' for plain objects and non-message arrays", () => { + expect(detectFieldKind({a: 1})).toBe("object") + expect(detectFieldKind(["a", "b"])).toBe("object") + expect(detectFieldKind([1, 2, 3])).toBe("object") + }) + + it("returns 'chat' for arrays of role-tagged messages (overrides 'object')", () => { + expect( + detectFieldKind([ + {role: "user", content: "hi"}, + {role: "assistant", content: "hello"}, + ]), + ).toBe("chat") + }) + + it("keeps a JSON-shaped string as 'string' (gap-04 invariant)", () => { + // The chip says what the value IS, not what it looks like. + // A string that contains JSON text is still a string. + expect(detectFieldKind('{"a":1}')).toBe("string") + }) +}) + +describe("view-types: detectNestedKind (precise nested 6-way)", () => { + it("distinguishes string / number / boolean / null", () => { + expect(detectNestedKind("x")).toBe("string") + expect(detectNestedKind(0)).toBe("number") + expect(detectNestedKind(true)).toBe("boolean") + expect(detectNestedKind(null)).toBe("null") + }) + + it("distinguishes object from array (unlike detectFieldKind)", () => { + expect(detectNestedKind({a: 1})).toBe("object") + expect(detectNestedKind([1, 2])).toBe("array") + }) + + it("treats undefined as 'string' (matches form-widget fallback)", () => { + // Undefined doesn't have its own widget; defaulting to string lets + // the Input handle it gracefully. + expect(detectNestedKind(undefined)).toBe("string") + }) +}) + +describe("view-types: getViewOptions (per-value dropdown options)", () => { + it("offers Text / Markdown / JSON / YAML for strings — Text is default", () => { + const opts = getViewOptions("hello") + const values = opts.map((o) => o.value) + expect(values).toEqual(["text", "markdown", "json", "yaml"]) + expect(opts[0].hint).toBe("default") + }) + + it("offers Text / JSON / YAML for booleans — Text is default", () => { + const opts = getViewOptions(true) + expect(opts.map((o) => o.value)).toEqual(["text", "json", "yaml"]) + expect(opts[0].hint).toBe("default") + }) + + it("offers Form / JSON / YAML for objects — Form is default", () => { + const opts = getViewOptions({a: 1}) + expect(opts.map((o) => o.value)).toEqual(["form", "json", "yaml"]) + expect(opts[0].hint).toBe("default") + }) + + it("offers Form / JSON / YAML for non-message arrays — Form is default", () => { + const opts = getViewOptions(["a", "b"]) + expect(opts.map((o) => o.value)).toEqual(["form", "json", "yaml"]) + expect(opts[0].hint).toBe("default") + }) + + it("offers Chat / JSON / YAML for role-tagged messages arrays — Chat is default", () => { + const opts = getViewOptions([ + {role: "user", content: "hi"}, + {role: "assistant", content: "hello"}, + ]) + expect(opts.map((o) => o.value)).toEqual(["chat", "json", "yaml"]) + expect(opts[0].hint).toBe("default") + }) + + it("always includes JSON and YAML as fallback options", () => { + for (const value of ["x", 42, true, null, {a: 1}, [1, 2]]) { + const values = getViewOptions(value).map((o) => o.value) + expect(values).toContain("json") + expect(values).toContain("yaml") + } + }) +}) + +describe("view-types: getDefaultViewForValue", () => { + it("returns the first option from getViewOptions for known kinds", () => { + expect(getDefaultViewForValue("hello")).toBe("text") + expect(getDefaultViewForValue(true)).toBe("text") + expect(getDefaultViewForValue({a: 1})).toBe("form") + expect(getDefaultViewForValue([1])).toBe("form") + expect(getDefaultViewForValue([{role: "user", content: "x"}])).toBe("chat") + }) + + it("treats undefined as a string-kind value (default → 'text')", () => { + expect(getDefaultViewForValue(undefined)).toBe("text") + }) +}) + +describe("view-types: expected-type-aware variants", () => { + it("falls back to expectedType when value is undefined", () => { + // Draft `geo` port referenced via `{{geo.region}}` → object port. + expect(getDefaultViewForExpectedType(undefined, "object")).toBe("form") + // Arrays default to JSON instead of Form — empty arrays have no + // add-item affordance in Form view. See the dedicated test below. + expect(getDefaultViewForExpectedType(undefined, "array")).toBe("json") + expect(getDefaultViewForExpectedType(undefined, "boolean")).toBe("text") + expect(getDefaultViewForExpectedType(undefined, "number")).toBe("text") + expect(getDefaultViewForExpectedType(undefined, "string")).toBe("text") + }) + + it("falls back when value is null or empty string", () => { + expect(getDefaultViewForExpectedType(null, "object")).toBe("form") + expect(getDefaultViewForExpectedType("", "object")).toBe("form") + }) + + it("ignores expectedType when value is non-empty (value drives the kind)", () => { + // Real string value beats `expectedType: "object"` — the runtime + // value is the source of truth once it exists. + expect(getDefaultViewForExpectedType("hello", "object")).toBe("text") + // Real chat array beats `expectedType: "object"`. + const chat = [{role: "user", content: "hi"}] + expect(getDefaultViewForExpectedType(chat, "object")).toBe("chat") + }) + + it("returns the same options the value-driven helper would for empty + unknown type", () => { + // No expectedType → falls all the way through to value-driven defaults. + expect(getDefaultViewForExpectedType(undefined, undefined)).toBe("text") + }) + + it("getViewOptionsForExpectedType offers Form first for object drafts", () => { + const opts = getViewOptionsForExpectedType(undefined, "object") + expect(opts[0]?.value).toBe("form") + expect(opts.map((o) => o.value)).toEqual(expect.arrayContaining(["json", "yaml"])) + }) + + it("getViewOptionsForExpectedType offers Text first for string drafts", () => { + const opts = getViewOptionsForExpectedType(undefined, "string") + expect(opts[0]?.value).toBe("text") + expect(opts.map((o) => o.value)).toEqual(expect.arrayContaining(["markdown", "json"])) + }) + + it("array drafts default to JSON (not Form) — Form has no add-item affordance", () => { + // Empty arrays in Form view show "(empty object)" with no way to + // add items. JSON's `[]` buffer is the more useful entry point. + // BUT: list ordering keeps Form first (kind-specific), JSON/YAML + // at the bottom — same convention as every other kind. Default + // mode is decoupled from list order. + expect(getDefaultViewForExpectedType(undefined, "array")).toBe("json") + const opts = getViewOptionsForExpectedType(undefined, "array") + expect(opts.map((o) => o.value)).toEqual(["form", "json", "yaml"]) + }) + + it("array drafts switch to value-driven options once a real array exists", () => { + // Real array → value-driven path: Form is default for objects/arrays + // with items (FormView renders them per-index). + const arr = ["en", "fr"] + expect(getDefaultViewForExpectedType(arr, "array")).toBe("form") + }) + + it("keeps JSON / YAML at the bottom for every expectedType (consistency)", () => { + // The dropdown should read the same regardless of which type the + // draft is — kind-specific modes first, JSON then YAML at the end. + const string = getViewOptionsForExpectedType(undefined, "string") + expect(string.map((o) => o.value)).toEqual(["text", "markdown", "json", "yaml"]) + + const object = getViewOptionsForExpectedType(undefined, "object") + expect(object.map((o) => o.value)).toEqual(["form", "json", "yaml"]) + + const array = getViewOptionsForExpectedType(undefined, "array") + expect(array.map((o) => o.value)).toEqual(["form", "json", "yaml"]) + + const boolean = getViewOptionsForExpectedType(undefined, "boolean") + expect(boolean.map((o) => o.value)).toEqual(["text", "json", "yaml"]) + }) +}) + +describe("view-types: buildEmptyShapeFromSchema", () => { + it("returns null for null / non-object input", () => { + expect(buildEmptyShapeFromSchema(null)).toBeNull() + expect(buildEmptyShapeFromSchema(undefined)).toBeNull() + expect(buildEmptyShapeFromSchema("string")).toBeNull() + }) + + it("returns null for primitive schemas", () => { + expect(buildEmptyShapeFromSchema({type: "string"})).toBeNull() + expect(buildEmptyShapeFromSchema({type: "number"})).toBeNull() + expect(buildEmptyShapeFromSchema({type: "boolean"})).toBeNull() + }) + + it("returns an empty array for array schemas", () => { + expect(buildEmptyShapeFromSchema({type: "array"})).toEqual([]) + }) + + it("builds an empty-value object from flat properties", () => { + const schema = { + type: "object", + properties: {region: {type: "string"}, subregion: {type: "string"}}, + } + expect(buildEmptyShapeFromSchema(schema)).toEqual({region: "", subregion: ""}) + }) + + it("recursively builds nested shapes from nested object properties", () => { + const schema = { + type: "object", + properties: { + region: {type: "string"}, + coordinates: { + type: "object", + properties: {lat: {type: "string"}, lng: {type: "string"}}, + }, + }, + } + expect(buildEmptyShapeFromSchema(schema)).toEqual({ + region: "", + coordinates: {lat: "", lng: ""}, + }) + }) + + it("prefers _pathHints over flat properties when both are present", () => { + // Playground's `buildSubPathSchema` flattens nested sub-paths into + // top-level `{type: "string"}` properties but preserves the original + // sub-paths in `_pathHints`. The helper reconstructs the nesting + // from `_pathHints` to surface the right structure. + const schema = { + type: "object", + properties: { + region: {type: "string"}, + subregion: {type: "string"}, + coordinates: {type: "string"}, // flattened + }, + _pathHints: ["region", "subregion", "coordinates.lat", "coordinates.lng"], + } + expect(buildEmptyShapeFromSchema(schema)).toEqual({ + region: "", + subregion: "", + coordinates: {lat: "", lng: ""}, + }) + }) + + it("handles empty _pathHints gracefully (falls back to properties)", () => { + const schema = { + type: "object", + properties: {region: {type: "string"}}, + _pathHints: [], + } + expect(buildEmptyShapeFromSchema(schema)).toEqual({region: ""}) + }) + + it("returns an empty object for object schemas with no properties / no hints", () => { + // Type-is-object but nothing to seed. The Form view will render an + // empty form (no fields) — caller can interpret that as "no shape". + expect(buildEmptyShapeFromSchema({type: "object"})).toBeNull() + }) + + it("doesn't crash on malformed _pathHints (non-string entries)", () => { + const schema = { + type: "object", + // @ts-expect-error — testing runtime robustness + _pathHints: ["valid.path", null, 42, "another"], + } + // Defensive — the helper just skips non-string entries. + const result = buildEmptyShapeFromSchema(schema) + expect(result).toEqual( + expect.objectContaining({ + valid: {path: ""}, + another: "", + }), + ) + }) +}) diff --git a/web/packages/agenta-playground-ui/package.json b/web/packages/agenta-playground-ui/package.json index dc76c13099..622709009d 100644 --- a/web/packages/agenta-playground-ui/package.json +++ b/web/packages/agenta-playground-ui/package.json @@ -28,6 +28,7 @@ "./comparison-view": "./src/components/ExecutionItemComparisonView/index.tsx", "./execution-items": "./src/components/ExecutionItems/index.tsx", "./execution-item-comparison-view": "./src/components/ExecutionItemComparisonView/index.tsx", + "./playground-inputs-body": "./src/components/PlaygroundInputsBody/index.tsx", "./workflow-revision-drawer": "./src/components/WorkflowRevisionDrawer/index.ts" }, "dependencies": { diff --git a/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/ComparisonLayout.tsx b/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/ComparisonLayout.tsx index 5695931cab..3ebe00a880 100644 --- a/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/ComparisonLayout.tsx +++ b/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/ComparisonLayout.tsx @@ -1,6 +1,7 @@ import React, {useCallback, useMemo} from "react" import type {PlaygroundNode} from "@agenta/entities/runnable" +import {workflowMolecule} from "@agenta/entities/workflow" import {executionItemController, playgroundController} from "@agenta/playground" import {getEvaluatorVerdictFromOutput} from "@agenta/playground/utils" import type {DropdownButtonOption, DropdownButtonOptionStatus} from "@agenta/ui/components" @@ -16,7 +17,11 @@ import clsx from "clsx" import {atom, useAtomValue, useSetAtom} from "jotai" import {VariableControlAdapter} from "@agenta/playground-ui/adapters" -import {openPlaygroundFocusDrawerAtom} from "@agenta/playground-ui/state" +import {PlaygroundInputsBodyHost} from "@agenta/playground-ui/playground-inputs-body" +import { + openPlaygroundFocusDrawerAtom, + useNewPlaygroundInputsBodyAtom, +} from "@agenta/playground-ui/state" import {usePlaygroundUIOptional} from "../../../../context/PlaygroundUIContext" @@ -100,6 +105,53 @@ const ComparisonLayout = ({ const structuralRootNode = rootNodes[0] ?? null const hasDownstreamNodes = downstreamNodes.length > 0 + // Downstream key — same shape as SingleLayout. Used by + // `PlaygroundInputsBodyHost`'s visibility selector to namespace the + // referenced-vs-unreferenced split per evaluator chain. + const downstreamKey = useMemo( + () => + downstreamNodes + .map((n) => n.entityId) + .sort() + .join(","), + [downstreamNodes], + ) + + // Feature flag — when true, the comparison view renders a single + // shared `PlaygroundInputsBodyHost` (V2 bordered cards + type chips + + // "View as ▾" dropdown) instead of the per-variable + // `VariableControlAdapter` loop. Off by default; OSS opts in. + const useNewInputsBody = useAtomValue(useNewPlaygroundInputsBodyAtom) + + // Resolve the active prompt template_format from the structural root + // (the first depth-0 node). In comparison view multiple variants may + // declare different formats — we pick the structural root's value as + // the canonical one for tokenization inside chat-mode variable inputs. + // The trade-off is acceptable: the same testcase data feeds all + // variants, and the chat editor's `templateFormat` only affects how + // `{{...}}` segments TOKENIZE inside message content (it doesn't + // change rendering at runtime). + const primaryEntityIdForTemplateFormat = structuralRootNode?.entityId ?? entityId ?? "" + const primaryWorkflowData = useAtomValue( + useMemo( + () => workflowMolecule.selectors.data(primaryEntityIdForTemplateFormat), + [primaryEntityIdForTemplateFormat], + ), + ) + const promptTemplateFormat = useMemo<"mustache" | "curly" | "fstring" | "jinja2">(() => { + const params = primaryWorkflowData?.data?.parameters as Record | undefined + const prompt = params?.prompt as Record | undefined + const raw = + (prompt?.template_format as string | undefined) ?? + (prompt?.templateFormat as string | undefined) ?? + (params?.template_format as string | undefined) ?? + (params?.templateFormat as string | undefined) + if (raw === "mustache") return "mustache" + if (raw === "jinja2" || raw === "jinja") return "jinja2" + if (raw === "fstring") return "fstring" + return "curly" + }, [primaryWorkflowData?.data?.parameters]) + const {getNodeLabel} = usePlaygroundNodeLabels(nodes) const mapStatuses = useCallback( @@ -240,80 +292,122 @@ const ComparisonLayout = ({ >
- {variableIds.map((variableId, index) => ( -
- + {/* Row-level controls — moved out of the + * per-variable header cluster in the new + * inputs body. Open focus drawer + delete + * row live together in a single toolbar + * above the shared inputs body. */} + {!inputOnly && ( +
+ } + onClick={() => + openFocusDrawer({ + rowId, + entityId: + structuralRootNode?.entityId ?? entityId, + }) + } + disabled={!(structuralRootNode?.entityId ?? entityId)} + tooltipProps={{title: "Open details"}} + /> + } + onClick={() => deleteRow(rowId)} + disabled={rowCount <= 1} + tooltipProps={{title: "Remove"}} + /> +
+ )} + + + ) : ( + variableIds.map((variableId, index) => ( +
- - {index === 0 ? ( + > + + + {index === 0 ? ( + + } + onClick={() => + openFocusDrawer({ + rowId, + entityId: + structuralRootNode?.entityId ?? + entityId, + }) + } + disabled={ + !( + structuralRootNode?.entityId ?? + entityId + ) + } + tooltipProps={{title: "Open details"}} + /> + ) : null} - } - onClick={() => - openFocusDrawer({ - rowId, - entityId: - structuralRootNode?.entityId ?? - entityId, - }) - } - disabled={ - !( - structuralRootNode?.entityId ?? - entityId - ) - } - tooltipProps={{title: "Open details"}} + icon={} + onClick={() => deleteRow(rowId)} + disabled={rowCount <= 1} + tooltipProps={{title: "Remove"}} /> - ) : null} - } - onClick={() => deleteRow(rowId)} - disabled={rowCount <= 1} - tooltipProps={{title: "Remove"}} - /> - - ) : undefined - } - /> -
- ))} + + ) : undefined + } + /> +
+ )) + )}
diff --git a/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/SingleLayout.tsx b/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/SingleLayout.tsx index 7df0f3d29a..4826bcf081 100644 --- a/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/SingleLayout.tsx +++ b/web/packages/agenta-playground-ui/src/components/ExecutionItems/assets/ExecutionRow/SingleLayout.tsx @@ -27,7 +27,11 @@ import {useAtom, useAtomValue, useSetAtom} from "jotai" import {atomWithStorage} from "jotai/utils" import {VariableControlAdapter} from "@agenta/playground-ui/adapters" -import {openPlaygroundFocusDrawerAtom} from "@agenta/playground-ui/state" +import {PlaygroundInputsBodyHost} from "@agenta/playground-ui/playground-inputs-body" +import { + openPlaygroundFocusDrawerAtom, + useNewPlaygroundInputsBodyAtom, +} from "@agenta/playground-ui/state" import {usePlaygroundUIOptional} from "../../../../context/PlaygroundUIContext" import {useRepetitionResult} from "../../../../hooks/useRepetitionResult" @@ -618,6 +622,25 @@ const SingleView = ({ useMemo(() => workflowMolecule.selectors.isDirty(entityId), [entityId]), ) + // Resolve the active prompt template_format from the primary entity's + // parameters. Passed into `PlaygroundInputsBodyHost` so chat-mode + // variable inputs tokenize `{{...}}` with the right rules (mustache + // sections, jinja2, etc). Falls back to `"curly"` to match the + // editor stack's default when no value is stored yet. + const promptTemplateFormat = useMemo<"mustache" | "curly" | "fstring" | "jinja2">(() => { + const params = primaryWorkflowData?.data?.parameters as Record | undefined + const prompt = params?.prompt as Record | undefined + const raw = + (prompt?.template_format as string | undefined) ?? + (prompt?.templateFormat as string | undefined) ?? + (params?.template_format as string | undefined) ?? + (params?.templateFormat as string | undefined) + if (raw === "mustache") return "mustache" + if (raw === "jinja2" || raw === "jinja") return "jinja2" + if (raw === "fstring") return "fstring" + return "curly" + }, [primaryWorkflowData?.data?.parameters]) + const executionRowIds = useAtomValue( executionItemController.selectors.executionRowIds, ) as string[] @@ -673,6 +696,11 @@ const SingleView = ({ const isExecutionExpanded = inputOnly || !isCollapsed const isWaitingForVariableControls = variableIds.length === 0 && (schemaInputKeys.length > 0 || Boolean(runnableQuery.isPending)) + + // Feature flag — when true, the non-grouped (flat) variable list renders + // through `PlaygroundInputsBodyHost` (V2-aligned bordered cards with + // type chips + "View as ▾" dropdown). Off by default; OSS opts in. + const useNewInputsBody = useAtomValue(useNewPlaygroundInputsBodyAtom) const collapseDurationMs = hasInteractedWithCollapse ? 300 : 0 useEffect(() => { @@ -865,27 +893,69 @@ const SingleView = ({ ) } - // Flat layout — apps, and evaluators with no - // extracted field ports (default template). + // New playground inputs body — V2-aligned + // bordered cards with type chips + "View + // as ▾" dropdowns. Behind a feature flag + // so the existing per-variable layout stays + // default until the new UX is signed off. + if (useNewInputsBody) { + // Grouped layout — evaluators with + // extracted field ports. Field ports + + // the `inputs` envelope catch-all share + // `data.inputs` at runtime; the new + // inputs body renders both inside the + // same left-border "inputs" section, + // and the `outputs` envelope in its + // own section. The section blocks come + // through the host's `sections` prop. + if (useGroupedLayout) { + const groupedSections = [ + { + ariaLabel: "inputs", + variableNames: [ + ...fieldPortIds, + ...(hasInputsEnvelope ? ["inputs"] : []), + ], + }, + ...(hasOutputsEnvelope + ? [ + { + ariaLabel: "outputs", + variableNames: ["outputs"], + }, + ] + : []), + ] + return ( + + ) + } + return ( + + ) + } + + // Legacy path (flag off) — flat layout for + // apps and evaluators-with-no-fields, grouped + // legacy SectionBlock layout for evaluators + // with extracted field ports. Each port renders + // through the per-variable + // `VariableControlAdapter` with its full header + // (label + ⓘ tooltip + hover-revealed actions). if (!useGroupedLayout) { return variableIds.map((id) => renderVariable(id)) } - - // Grouped layout — evaluators with field - // ports. Field ports + the envelope catch-all - // share `data.inputs` at runtime, so we wrap - // them in one left-border block to surface - // that relationship visually. Each port - // renders with its full header (label + ⓘ - // tooltip + hover-revealed action cluster: - // JSON/text toggle, markdown, copy, collapse) - // and the editor's own border, so envelope - // and field rows look identical and the - // hover affordances stay intact. The group - // identity comes from the container; we - // intentionally don't add a separate header - // label to avoid colliding with the envelope - // port's own header. return ( <> diff --git a/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/PlaygroundInputsBody.tsx b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/PlaygroundInputsBody.tsx new file mode 100644 index 0000000000..9d2641325e --- /dev/null +++ b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/PlaygroundInputsBody.tsx @@ -0,0 +1,210 @@ +/** + * PlaygroundInputsBody — the inputs panel inside one playground generation + * card. + * + * Replaces the per-variable SharedEditor cells the playground used to render + * via `VariableControlAdapter`. Instead, each variable gets its own bordered + * card with: + * - type chip (granular, via #4394's TypeChip + inferLogicalType) + * - "View as ▾" dropdown (text / markdown / chat / form / json / yaml, + * scoped to what makes sense for the value's kind) + * - per-view body (SharedEditor / FormView / ChatMessageList / etc.) + * + * The component is presentational + atom-aware (per-variable view mode is + * an atom family keyed by `(rowId, varName)` — see `viewModeAtoms.ts`). All + * write-backs flow as NATIVE values via `onValueChange`. No stringification + * on the way out (RFC: "native JSON stays native until template rendering"). + * + * Designed to be wired into the existing OSS playground in Step 6 — this + * branch ships the component and its primitives; Step 6 swaps the current + * `VariableControlAdapter` rendering for ``. + * + * Visibility rule (computed by the parent, passed in here): + * - `inputs`: referenced variables, including draft ones. + * Each gets an expanded card. + * - `unreferencedColumns`: testcase columns the prompt doesn't reference. + * Collapsed under a single footer row. + * + * The parent owns the prompt-text → referenced-variables computation (lives + * in OSS / agenta-playground state), so this component stays pure. + */ + +import type {ExpectedType, ViewType} from "@agenta/entity-ui/view-types" +import { + getDefaultViewForExpectedType, + getViewOptionsForExpectedType, +} from "@agenta/entity-ui/view-types" + +import {UnreferencedColumnsFooter} from "./UnreferencedColumnsFooter" +import {VariableCard} from "./VariableCard" + +export interface PlaygroundInputsBodyVariable { + /** Variable name (testcase column or template-referenced variable). */ + name: string + /** Native value, or `undefined` for draft variables. */ + value: unknown + /** True when the variable is referenced by the prompt but not authored + * on the testcase yet. Renders a `[draft]` badge. */ + isDraft?: boolean + /** Optional tooltip text explaining the variable's role. Surfaced as a + * small Info icon in the card header — used for evaluator envelope + * variables (`inputs`/`outputs`) to keep the legacy guidance visible. */ + helpText?: string + /** Declared port type from the runnable schema (`object` / `array` / + * `string` / `number` / `integer` / `boolean`). When the variable is + * a draft (no value yet), this drives the default view mode and the + * TypeChip variant so the card opens in the right shape — e.g. a + * `geo` port referenced via `{{geo.region}}` opens as Form with an + * `object` chip instead of a text input with a `null` chip. */ + expectedType?: ExpectedType + /** Declared port schema (JSON Schema fragment with `properties` / + * `_pathHints`). When the variable is a draft, Form / JSON / YAML + * modes pre-render an empty-value skeleton matching this shape so + * the user sees the expected sub-fields without having to add them + * manually. Render-only — the testcase value stays untouched until + * the user actually edits a field. */ + expectedSchema?: unknown +} + +/** + * Optional section grouping for the variable cards. When present (see + * `PlaygroundInputsBodyProps.sections` below), each section renders inside + * a left-border accent block — mirrors the legacy `` look the + * grouped evaluator layout used in `SingleLayout`. Variable cards inside a + * section behave exactly like ungrouped cards otherwise. + */ +export interface PlaygroundInputsBodySection { + /** Aria label for the group (e.g. `"inputs"` / `"outputs"`). Not + * rendered as a visible heading — the left-border + the per-card + * TypeChip + name carry the disambiguation. */ + ariaLabel: string + /** Variables rendered inside this section, in order. */ + variables: PlaygroundInputsBodyVariable[] +} + +export interface PlaygroundInputsBodyProps { + /** Stable identifier for the playground generation row this card lives + * in. Used to key per-variable view-mode atoms — must be stable across + * testcase column adds (so draft variables don't lose their selected + * mode when the column gets persisted). */ + rowId: string + /** Variables referenced by the prompt chain. Rendered as expanded cards + * in order. Include draft variables (referenced but not on testcase) + * with `isDraft: true`. Ignored when `sections` is provided. */ + inputs: PlaygroundInputsBodyVariable[] + /** Optional grouped layout. When present, replaces the flat `inputs` + * rendering with one left-border block per section. Used by the + * evaluator grouped layout (`inputs` envelope + extracted field ports + * in one block, `outputs` envelope in another). */ + sections?: PlaygroundInputsBodySection[] + /** Testcase columns NOT referenced by the prompt chain. Rendered under + * a single collapsed footer below all variable cards. Pass `undefined` + * or `[]` to skip the footer entirely. */ + unreferencedColumns?: PlaygroundInputsBodyVariable[] + /** Whether referenced variable cards are editable. */ + editable: boolean + /** When non-empty, every card shows a small database indicator in the + * header — `Synced from {name}`. Communicates that the row's data + * comes from a testset rather than being authored locally. */ + connectedSourceName?: string | null + /** Writes the new value for an existing column to the testcase store. + * Implementation should route through `testcaseMolecule.actions.update` + * so the testcase entity is updated atomically. NATIVE value — no + * stringification by the caller. */ + onValueChange: (name: string, value: unknown) => void + /** Optional. Creates a new testcase column on first edit of a draft + * variable. If undefined, draft edits route through `onValueChange` + * and the caller decides how to persist. */ + onAddDraftColumn?: (name: string, value: unknown) => void + /** Optional. Notified when the user changes the view mode for a card. */ + onViewModeChange?: (name: string, mode: ViewType) => void + /** Optional. When `unreferencedColumns` is shown and the footer is + * expanded, gate edits to those rows. Defaults to read-only. */ + unreferencedEditable?: boolean + /** Active prompt template format. Forwarded to every `VariableCard` + * so chat-mode rendering tokenizes the right `{{...}}` syntax. */ + templateFormat?: "mustache" | "curly" | "fstring" | "jinja2" +} + +export function PlaygroundInputsBody({ + rowId, + inputs, + sections, + unreferencedColumns, + editable, + onValueChange, + onAddDraftColumn, + onViewModeChange, + unreferencedEditable = false, + connectedSourceName, + templateFormat, +}: PlaygroundInputsBodyProps) { + // `sections` takes precedence over the flat `inputs` list. We still need + // to look up by name to route draft edits, so unify the membership + // source here. + const allVariables: PlaygroundInputsBodyVariable[] = sections + ? sections.flatMap((s) => s.variables) + : inputs + + const handleValueChange = (name: string, value: unknown) => { + const variable = allVariables.find((v) => v.name === name) + if (variable?.isDraft && onAddDraftColumn) { + onAddDraftColumn(name, value) + } else { + onValueChange(name, value) + } + } + + const renderCard = (variable: PlaygroundInputsBodyVariable) => ( + + ) + + return ( +
+ {sections + ? sections.map((section) => ( +
` accent in + // `SingleLayout`. No visible heading — the chip + name + // on each card carries the per-variable label, and the + // left-border conveys the group identity. + className="flex flex-col gap-2 pl-3 border-0 border-l-2 border-solid border-[#1677FF22]" + > + {section.variables.map(renderCard)} +
+ )) + : inputs.map(renderCard)} + {unreferencedColumns && unreferencedColumns.length > 0 ? ( + ({name: c.name, value: c.value}))} + editable={unreferencedEditable} + onValueChange={ + unreferencedEditable + ? (name, value) => onValueChange(name, value) + : undefined + } + /> + ) : null} +
+ ) +} diff --git a/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/PlaygroundInputsBodyHost.tsx b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/PlaygroundInputsBodyHost.tsx new file mode 100644 index 0000000000..d870d1abe3 --- /dev/null +++ b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/PlaygroundInputsBodyHost.tsx @@ -0,0 +1,196 @@ +/** + * PlaygroundInputsBodyHost — atom-aware wrapper around `PlaygroundInputsBody`. + * + * Bridges the playground execution state to the presentational component: + * - `inputs` + `unreferencedColumns` come from + * `executionItemController.selectors.inputsVisibility({testcaseId, downstreamKey})`. + * - `helpText` per variable comes from `inputPortSchemaMap` — used by the + * evaluator envelope variables (`inputs`/`outputs`) to keep the legacy + * guidance tooltip visible after the migration. + * - Edits flow to `executionItemController.actions.setTestcaseCellValue`, + * which writes through `testcaseMolecule.actions.update` so the testcase + * entity is the single source of truth. + * - Draft variables (referenced by prompt but absent from testcase) write + * through the SAME action — `setTestcaseCellValue` is happy to create a + * new column on first set, so we don't need a separate `onAddDraftColumn`. + * - Optional `sections` prop partitions visibility.inputs into named groups + * (left-border accent), used by the evaluator grouped layout in + * SingleLayout. + */ + +import {useCallback, useMemo} from "react" + +import {loadableController} from "@agenta/entities/runnable" +import {executionItemController, playgroundController} from "@agenta/playground" +import {atom, useAtomValue, useSetAtom} from "jotai" + +import {PlaygroundInputsBody} from "./PlaygroundInputsBody" +import type { + PlaygroundInputsBodySection, + PlaygroundInputsBodyVariable, +} from "./PlaygroundInputsBody" + +export interface PlaygroundInputsBodyHostProps { + /** Testcase row ID (also the playground generation row ID — they're + * the same in the loadable routing per the testcaseMolecule contract). */ + rowId: string + /** Downstream key used by the visibility selector to namespace its + * computation per-evaluator-chain. Should match the key the caller + * uses with `variableKeysForDownstream`. */ + downstreamKey: string + /** Whether the cards are editable (vs read-only). */ + editable: boolean + /** Optional grouped layout. Each entry pulls the named variables out of + * `visibility.inputs` into a dedicated section. Variables NOT listed in + * any section stay in their original order under no group block; in + * practice the caller lists every referenced key so this is rare. + * Order of sections is preserved as-passed. */ + sections?: { + ariaLabel: string + variableNames: string[] + }[] + /** Active prompt template format. Forwarded to every variable card so + * chat-mode rendering tokenizes the right `{{...}}` syntax. The + * caller resolves this from the primary entity's prompt config — the + * host doesn't read it itself because it has no single canonical + * entity in comparison layouts (multiple variants may differ). */ + templateFormat?: "mustache" | "curly" | "fstring" | "jinja2" +} + +export function PlaygroundInputsBodyHost({ + rowId, + downstreamKey, + editable, + sections, + templateFormat, +}: PlaygroundInputsBodyHostProps) { + const visibility = useAtomValue( + useMemo( + () => + executionItemController.selectors.inputsVisibility({ + testcaseId: rowId, + downstreamKey, + }), + [rowId, downstreamKey], + ), + ) + + // Read the input port schema map so we can inject helpText + declared + // port type + JSON-schema fragment onto each variable entry: + // - `helpText` → evaluator envelope variables (`inputs`/`outputs`) + // keep the legacy guidance tooltip. + // - `expectedType` → drives the default view mode + TypeChip for + // DRAFT variables (no value yet). Without it, a + // `geo` port referenced via `{{geo.region}}` + // opens as a text input with a `null` chip + // instead of Form + `object` chip. + // - `expectedSchema` → seeds Form / JSON / YAML modes on drafts with an + // empty-value skeleton matching the expected + // sub-fields (so `geo` shows `region`, `subregion`, + // `coordinates.lat/lng` before the user types). + const portSchemaMap = useAtomValue( + executionItemController.selectors.inputPortSchemaMap, + ) as Record + + const enrichedInputs = useMemo( + () => + visibility.inputs.map((v) => { + const portSchema = portSchemaMap[v.name] + const help = portSchema?.helpText + const type = portSchema?.type as PlaygroundInputsBodyVariable["expectedType"] + const schema = portSchema?.schema + if (!help && !type && !schema) return v + return { + ...v, + ...(help ? {helpText: help} : {}), + ...(type ? {expectedType: type} : {}), + ...(schema ? {expectedSchema: schema} : {}), + } + }), + [visibility.inputs, portSchemaMap], + ) + + // Partition enriched inputs into sections when `sections` is provided. + // Variables not listed in any section are appended as ungrouped at the + // end (caller responsibility to list every key for a clean layout). + const bodySections = useMemo(() => { + if (!sections) return undefined + const byName = new Map(enrichedInputs.map((v) => [v.name, v])) + const claimed = new Set() + const groups: PlaygroundInputsBodySection[] = sections.map((spec) => { + const variables: PlaygroundInputsBodyVariable[] = [] + for (const name of spec.variableNames) { + const v = byName.get(name) + if (!v) continue + variables.push(v) + claimed.add(name) + } + return {ariaLabel: spec.ariaLabel, variables} + }) + const leftover = enrichedInputs.filter((v) => !claimed.has(v.name)) + if (leftover.length > 0) { + groups.push({ariaLabel: "other", variables: leftover}) + } + return groups + }, [sections, enrichedInputs]) + + const setCellValue = useSetAtom(executionItemController.actions.setTestcaseCellValue) + + // Connected-source name (set when the row is sourced from a testset + // rather than authored locally). Every card uses this to surface the + // unified database indicator — same per-card, gated by the global + // loadable state. + const loadableId = useAtomValue( + useMemo(() => playgroundController.selectors.loadableId(), []), + ) as string | null + // Build a single read-only atom that returns the connected-source + // descriptor (or null when no loadable is mounted). The cast through + // `unknown` keeps the conditional atom typeable — the runtime always + // returns the `{id, name, type}` shape (or null), but the atom-family + // factory has its own readonly atom type that wouldn't normally union + // cleanly with a fallback `atom(() => null)`. + const connectedSourceAtom = useMemo( + () => + loadableId + ? loadableController.selectors.connectedSource(loadableId) + : (atom(() => null) as unknown as ReturnType< + typeof loadableController.selectors.connectedSource + >), + [loadableId], + ) + const connectedSource = useAtomValue(connectedSourceAtom) as { + id: string | null + name: string | null + } | null + const connectedSourceName = connectedSource?.id ? (connectedSource.name ?? null) : null + + const handleValueChange = useCallback( + (name: string, value: unknown) => { + setCellValue({testcaseId: rowId, column: name, value}) + }, + [setCellValue, rowId], + ) + + return ( + + ) +} diff --git a/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/UnreferencedColumnsFooter.tsx b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/UnreferencedColumnsFooter.tsx new file mode 100644 index 0000000000..18defc6f1d --- /dev/null +++ b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/UnreferencedColumnsFooter.tsx @@ -0,0 +1,88 @@ +/** + * UnreferencedColumnsFooter — collapsed-by-default footer rendered once + * below all variable cards. + * + * "N unused testcase columns hidden because the prompt does not reference + * them." Click to expand and render the unreferenced columns as collapsed + * read-only cards beneath. + * + * Rendered ONCE per generation card (not per variable). When the prompt + * adds a reference to a previously-unused column, the parent moves that + * entry from `unreferencedColumns` to `inputs` and the footer count drops + * by one — no special handling here. + */ + +import {useState} from "react" + +import {getDefaultViewForValue, getViewOptions} from "@agenta/entity-ui/view-types" +import {CaretDown, CaretRight} from "@phosphor-icons/react" +import {Button} from "antd" + +import {VariableCard} from "./VariableCard" + +interface UnreferencedColumn { + name: string + value: unknown +} + +interface UnreferencedColumnsFooterProps { + /** Stable identifier for the generation row this footer lives in. */ + rowId: string + columns: UnreferencedColumn[] + /** Whether expanded cards are editable. Most surfaces will pass `false` + * here — unused columns shouldn't tempt the user into editing them. + * Set `true` if the parent wants edits to be allowed once revealed. */ + editable?: boolean + /** Fires when the user edits a previously-unused column. Most callers + * will leave this undefined (read-only). */ + onValueChange?: (name: string, value: unknown) => void +} + +export function UnreferencedColumnsFooter({ + rowId, + columns, + editable = false, + onValueChange, +}: UnreferencedColumnsFooterProps) { + const [expanded, setExpanded] = useState(false) + + if (columns.length === 0) return null + + const summary = `${columns.length} unused testcase column${columns.length === 1 ? "" : "s"} hidden because the prompt does not reference them.` + + return ( +
+ + {expanded ? ( +
+ {columns.map((col) => ( + { + /* no-op when parent didn't supply a handler */ + }) + } + /> + ))} +
+ ) : null} +
+ ) +} diff --git a/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/VariableCard.tsx b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/VariableCard.tsx new file mode 100644 index 0000000000..2b74c6718b --- /dev/null +++ b/web/packages/agenta-playground-ui/src/components/PlaygroundInputsBody/VariableCard.tsx @@ -0,0 +1,522 @@ +/** + * VariableCard — a single bordered input card for one playground variable. + * + * Header (single line): + * - Left: variable name (mono, blue), TypeChip (inferLogicalType + + * chat-detection override), optional [draft] badge. + * - Right: ViewTypeSelect (the "View as ▾" dropdown — Text/Markdown/Chat/ + * Form/JSON/YAML, options vary per kind). + * + * Body switches by the active view mode: + * - text → Text editor (string), antd InputNumber (number), Switch + * (boolean), "null" placeholder (null) + * - markdown → SharedEditor with markdownView enabled + * - chat → ChatMessageList over a messages array + * - form → FormView (recursive object/array editor) + * - json → SharedEditor (codeOnly language="json"), parse-on-edit + * - yaml → SharedEditor (codeOnly language="yaml"), parse-on-edit + * + * All edits write NATIVE values via `onValueChange(name, value)` — the card + * never stringifies on the way out (RFC: "native JSON stays native until + * template rendering"). The runtime gets objects as objects, arrays as + * arrays, numbers as numbers, etc. + */ + +import {useCallback, useEffect, useMemo, useState, type ReactNode} from "react" + +import { + FormView, + ViewTypeSelect, + buildEmptyShapeFromSchema, + coerceTextEdit, + inferLogicalType, + isChatMessagesArray, + parseJsonEdit, + parseYamlEdit, + valueToDisplay, +} from "@agenta/entity-ui/view-types" +import type {ExpectedType, LogicalType, ViewOption, ViewType} from "@agenta/entity-ui/view-types" +import {ChatMessageList} from "@agenta/ui/chat-message" +import type {SimpleChatMessage} from "@agenta/ui/chat-message" +import {SharedEditor} from "@agenta/ui/shared-editor" +import {TypeChip} from "@agenta/ui/type-chip" +import type {ChipVariant} from "@agenta/ui/type-chip" +import {CopySimple, Database, Info} from "@phosphor-icons/react" +import {Button, Input, InputNumber, Switch, Tag, Tooltip, Typography, message} from "antd" +import clsx from "clsx" +import {useAtom} from "jotai" + +import {variableViewModeAtomFamily} from "./viewModeAtoms" + +const {TextArea} = Input + +const {Text: AntText} = Typography + +interface VariableCardProps { + /** Stable identifier for the generation row this variable lives in. */ + rowId: string + /** Variable name (testcase column or referenced template variable). */ + name: string + /** Native value, or `undefined` for a draft variable. */ + value: unknown + /** Computed dropdown options for the value. Provided by the parent so we + * recompute consistently with how the parent decided which cards to + * render (e.g. for chat-shaped messages → Chat is offered). */ + options: ViewOption[] + /** The default view mode for this value. Used when the user hasn't + * explicitly chosen one yet (atom value is `null`). */ + defaultMode: ViewType + /** True when the variable is referenced by the prompt but not authored + * on the testcase yet. Renders a `[draft]` badge. */ + isDraft?: boolean + /** Optional tooltip text explaining what the variable represents. + * Surfaced as a small Info icon next to the name — matches the legacy + * `VariableControlAdapter` header treatment for evaluator envelope + * variables (`inputs`/`outputs`). */ + helpText?: string + /** Declared port type from the runnable schema. Used as the TypeChip + * fallback when the runtime value is empty (`undefined` / `null` / `""`) + * so a draft variable known to be an object shows the `object` chip + * instead of `null`. */ + expectedType?: ExpectedType + /** Declared port schema (JSON Schema fragment with `properties` / + * `_pathHints`). When the variable is a draft (no value yet), Form / + * JSON / YAML modes seed their initial render with an empty-value + * skeleton built from this schema, so the user sees the expected + * sub-fields without having to add them manually. Render-only — the + * testcase value stays untouched until the user actually edits. */ + expectedSchema?: unknown + /** When set, the card shows a small database indicator with a tooltip + * `Synced from {name}`. Communicates that this row's data comes from + * a testset rather than being authored locally. Unified across every + * card in the inputs body — the host either passes a name for all + * cards or none. */ + connectedSourceName?: string | null + /** Active prompt template format. When a value renders in chat mode, + * the inner `ChatMessageList`'s editors use this to tokenize + * `{{...}}` segments inside message content. Defaults to `"curly"` + * to match the rest of the editor stack's defaults. */ + templateFormat?: "mustache" | "curly" | "fstring" | "jinja2" + /** Whether the card is editable (vs read-only). */ + editable: boolean + /** Writes the new value to the testcase / draft store. NATIVE value. */ + onValueChange: (name: string, value: unknown) => void + /** Notified when the user picks a new view mode (optional — only the + * atom family is the source of truth; parents can subscribe here for + * side effects like analytics). */ + onViewModeChange?: (name: string, mode: ViewType) => void +} + +export function VariableCard({ + rowId, + name, + value, + options, + defaultMode, + isDraft, + helpText, + expectedType, + expectedSchema, + connectedSourceName, + templateFormat = "curly", + editable, + onValueChange, + onViewModeChange, +}: VariableCardProps) { + const [explicitMode, setExplicitMode] = useAtom( + variableViewModeAtomFamily({rowId, varName: name}), + ) + const mode: ViewType = explicitMode ?? defaultMode + + const handleModeChange = useCallback( + (next: ViewType) => { + setExplicitMode(next) + onViewModeChange?.(name, next) + }, + [setExplicitMode, onViewModeChange, name], + ) + + const handleValueChange = useCallback( + (next: unknown) => onValueChange(name, next), + [onValueChange, name], + ) + + const chipVariant = useMemo(() => { + if (isChatMessagesArray(value)) return "messages" + // For drafts (empty value), let the declared port type drive the + // chip so `geo` referenced as `{{geo.region}}` shows an `object` + // chip instead of falling through to `inferLogicalType(undefined) + // → "null"`. + const isEmpty = value === undefined || value === null || value === "" + if (isEmpty && expectedType) { + if (expectedType === "object") return "json-object" + if (expectedType === "array") return "json-array" + if (expectedType === "boolean") return "boolean" + if (expectedType === "number" || expectedType === "integer") return "number" + if (expectedType === "string") return "string" + } + return inferLogicalType(value) as ChipVariant + }, [value, expectedType]) + + // Render-only seed used by Form / JSON / YAML modes when the variable is + // a draft. The testcase value stays untouched — until the user actually + // edits a field, onChange never fires. See `buildEmptyShapeFromSchema` + // for the shape-derivation rules (prefers `_pathHints` over `properties`). + const seedShape = useMemo(() => { + if (!expectedSchema) return null + const isEmpty = value === undefined || value === null || value === "" + if (!isEmpty) return null + return buildEmptyShapeFromSchema(expectedSchema) + }, [value, expectedSchema]) + + // Copy the value as text. Primitives stringify naturally; structured + // values pretty-print as JSON. Drafts (no value yet) copy as empty + // string — defensive against undefined. + const handleCopy = useCallback(() => { + const text = + value === undefined || value === null + ? "" + : typeof value === "string" + ? value + : typeof value === "number" || typeof value === "boolean" + ? String(value) + : JSON.stringify(value, null, 2) + navigator.clipboard.writeText(text).then( + () => message.success({content: "Copied", duration: 1.5}), + () => message.error({content: "Copy failed", duration: 2}), + ) + }, [value]) + + return ( +
+
+
+ + {name} + + + {helpText ? ( + + + + ) : null} + {isDraft ? ( + + draft + + ) : null} +
+
+ {/* Unified action cluster — copy + (when connected) + * testset-sync indicator. Same set on every card so + * the row reads as a consistent block of inputs. */} + {connectedSourceName ? ( + + + + ) : null} + +
+
+
+ +
+
+ ) +} + +/* ── Body switcher ──────────────────────────────────────────────────── */ + +interface CardBodyProps { + mode: ViewType + value: unknown + /** Optional empty-value skeleton derived from the port schema, used as + * the render-only seed for Form / JSON / YAML modes when `value` is a + * draft (empty). The testcase stays untouched until the user actually + * edits a field — `onChange` only fires on real edits. */ + seedShape?: unknown + editable: boolean + onChange: (next: unknown) => void + /** Active prompt template format. Forwarded to `ChatMessageList` when + * the value renders in chat mode so its inner editors tokenize the + * right `{{...}}` syntax. */ + templateFormat?: "mustache" | "curly" | "fstring" | "jinja2" +} + +function CardBody({ + mode, + value, + seedShape, + editable, + onChange, + templateFormat = "curly", +}: CardBodyProps): ReactNode { + const originalType = useMemo(() => inferLogicalType(value), [value]) + + // For structured modes (form / json / yaml), prefer the schema-derived + // seed when the testcase value is still empty so the user sees the + // expected fields. Other modes (text / markdown / chat) get the raw + // value — seeding wouldn't help there. + const valueIsEmpty = value === undefined || value === null || value === "" + const renderValue = valueIsEmpty && seedShape != null ? seedShape : value + + if (mode === "form") { + // FormView expects an object record. If the value is an array, wrap + // its indexed children into a record { "0": ..., "1": ... } so the + // form can render. FormView itself recurses into arrays as well, + // but its root signature is `Record`. + const obj = + renderValue !== null && typeof renderValue === "object" && !Array.isArray(renderValue) + ? (renderValue as Record) + : Array.isArray(renderValue) + ? Object.fromEntries(renderValue.map((v, i) => [String(i), v])) + : {} + return ( + { + if (Array.isArray(renderValue)) { + // Recover an array from the indexed-record form. Sort + // the keys numerically and discard non-numeric keys + // (defensive — FormView preserves keys 1:1). + const rec = next as Record + const arr: unknown[] = [] + for (const [k, v] of Object.entries(rec)) { + const idx = Number(k) + if (Number.isInteger(idx) && idx >= 0) { + arr[idx] = v + } + } + onChange(arr) + } else { + onChange(next) + } + }} + /> + ) + } + + if (mode === "chat") { + const messages = isChatMessagesArray(value) ? (value as SimpleChatMessage[]) : [] + // Match the prop set used by `MessagesField` (drill-in drawer) which + // is the canonical working configuration for editing chat-shaped + // arrays. Without `enableTokens` / `templateFormat` the message + // editors mount their plugin stack in a constrained state and the + // content reads as static text. `allowFileUpload={false}` matches + // the variable-input UX — files belong in their own column, not + // inline in a testcase value. + return ( + onChange(next)} + disabled={!editable} + enableTokens + templateFormat={templateFormat} + allowFileUpload={false} + /> + ) + } + + if (mode === "json" || mode === "yaml") { + return ( + + ) + } + + // text / markdown for primitives — use the right widget per actual type. + // Borderless: the variable card itself supplies the encapsulating border, + // so the inner widget should NOT carry its own — otherwise the input + // visually "floats" inside the card and reads as a separate element. + if (originalType === "number" && mode === "text") { + return ( + onChange(next ?? null)} + placeholder="Enter number" + className="w-full max-w-[320px] !px-0" + /> + ) + } + + if (originalType === "boolean" && mode === "text") { + return ( + onChange(next)} + /> + ) + } + + // string + null fall through to a SharedEditor (also covers markdown). + return ( + + ) +} + +/* ── Text / Markdown editor ─────────────────────────────────────────── */ + +interface TextLeafEditorProps { + mode: ViewType // "text" | "markdown" only + value: unknown + editable: boolean + originalType: LogicalType + onChange: (next: unknown) => void +} + +function TextLeafEditor({mode, value, editable, originalType, onChange}: TextLeafEditorProps) { + const initial = useMemo(() => valueToDisplay(value, mode), [value, mode]) + const [buffer, setBuffer] = useState(initial) + + // Resync the local buffer when `value` changes externally — e.g. testcase + // refetch from the store, a row update from another surface, or a + // discard/revert that resets the cell. Without this the visible text + // goes stale while the underlying state has already moved on. + useEffect(() => { + setBuffer(initial) + }, [initial]) + + const handleChange = useCallback( + (next: string) => { + setBuffer(next) + onChange(coerceTextEdit(next, originalType)) + }, + [originalType, onChange], + ) + + // Plain TextArea with the borderless variant — no hover ring, no focus + // border, no padding, no shadow. The variable card supplies the only + // visible boundary; this editor melts into it so the label, controls, + // and value read as one block. + return ( +