Skip to content

feat: support per-test variable substitution in eval input #1254

@christso

Description

@christso

Summary

Add native eval-file support for per-test variables and template substitution in input / related test fields, similar to promptfoo's vars + {{name}} pattern.

Motivation

agentv import promptfoo currently has to eagerly render prompt templates into concrete strings because AgentV eval YAML appears to support only:

  • environment interpolation via ${{ VAR }}
  • grader prompt template variables like {{input}}, {{output}}, {{criteria}}

It does not appear to support promptfoo-style per-test variable substitution for eval inputs such as:

tests:
  - vars:
      question: What is the capital of France?
    input: "Answer clearly: {{question}}"

That makes imported suites less idiomatic and loses the original template structure.

Current behavior

AgentV eval files accept concrete input strings or message arrays. Environment interpolation works with ${{ VAR }}, but that is config/env-based, not per-test data-driven substitution.

Proposed direction

Support a minimal, explicit per-test templating model for eval files, for example:

tests:
  - vars:
      question: What is the capital of France?
    input: "Answer clearly: {{question}}"

Scope could include:

  • tests[].vars as a JSON/YAML object
  • substitution in input string shorthand and message content strings
  • maybe criteria / expected_output strings if that fits the model
  • preserve current ${{ ENV_VAR }} behavior separately

Why this matters

  • makes imported promptfoo suites cleaner
  • reduces duplication in eval YAML
  • enables more declarative eval authoring for humans and AI agents

Notes

This is distinct from:

  • environment interpolation (${{ VAR }})
  • llm-grader / composite prompt template variables ({{input}}, {{output}}, etc.)

Those already exist, but they are not a substitute for per-test input templating.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    In progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions