Summary
Add native eval-file support for per-test variables and template substitution in input / related test fields, similar to promptfoo's vars + {{name}} pattern.
Motivation
agentv import promptfoo currently has to eagerly render prompt templates into concrete strings because AgentV eval YAML appears to support only:
- environment interpolation via
${{ VAR }}
- grader prompt template variables like
{{input}}, {{output}}, {{criteria}}
It does not appear to support promptfoo-style per-test variable substitution for eval inputs such as:
tests:
- vars:
question: What is the capital of France?
input: "Answer clearly: {{question}}"
That makes imported suites less idiomatic and loses the original template structure.
Current behavior
AgentV eval files accept concrete input strings or message arrays. Environment interpolation works with ${{ VAR }}, but that is config/env-based, not per-test data-driven substitution.
Proposed direction
Support a minimal, explicit per-test templating model for eval files, for example:
tests:
- vars:
question: What is the capital of France?
input: "Answer clearly: {{question}}"
Scope could include:
tests[].vars as a JSON/YAML object
- substitution in
input string shorthand and message content strings
- maybe
criteria / expected_output strings if that fits the model
- preserve current
${{ ENV_VAR }} behavior separately
Why this matters
- makes imported promptfoo suites cleaner
- reduces duplication in eval YAML
- enables more declarative eval authoring for humans and AI agents
Notes
This is distinct from:
- environment interpolation (
${{ VAR }})
- llm-grader / composite prompt template variables (
{{input}}, {{output}}, etc.)
Those already exist, but they are not a substitute for per-test input templating.
Summary
Add native eval-file support for per-test variables and template substitution in
input/ related test fields, similar to promptfoo'svars+{{name}}pattern.Motivation
agentv import promptfoocurrently has to eagerly render prompt templates into concrete strings because AgentV eval YAML appears to support only:${{ VAR }}{{input}},{{output}},{{criteria}}It does not appear to support promptfoo-style per-test variable substitution for eval inputs such as:
That makes imported suites less idiomatic and loses the original template structure.
Current behavior
AgentV eval files accept concrete
inputstrings or message arrays. Environment interpolation works with${{ VAR }}, but that is config/env-based, not per-test data-driven substitution.Proposed direction
Support a minimal, explicit per-test templating model for eval files, for example:
Scope could include:
tests[].varsas a JSON/YAML objectinputstring shorthand and message content stringscriteria/expected_outputstrings if that fits the model${{ ENV_VAR }}behavior separatelyWhy this matters
Notes
This is distinct from:
${{ VAR }}){{input}},{{output}}, etc.)Those already exist, but they are not a substitute for per-test input templating.