Skip to content

pi-cli azure: empty assistant content in agent_end JSONL #922

@christso

Description

@christso

Summary

When using the pi-cli provider with subprovider: azure and azure-openai-responses, the pi CLI returns empty assistant content in the agent_end JSONL event. This causes all pi-cli azure evals to score 0.

Context

PR #921 added azure subprovider support for pi-coding-agent and pi-cli providers. The agentv connectivity, auth, and target resolution all work correctly. The pi CLI successfully connects to Azure, sends prompts, and the model generates responses. However, the response text is not populated in the JSONL output.

Reproduction

# targets.yaml
- name: pi-azure
  provider: pi-cli
  subprovider: azure
  base_url: ${{ AZURE_OPENAI_ENDPOINT }}  # resource name or URL
  model: ${{ AZURE_DEPLOYMENT_NAME }}
  api_key: ${{ AZURE_OPENAI_API_KEY }}
  grader_target: grader
# Run a simple eval
GRADER_TARGET=azure agentv eval run evals/self/azure-smoke.eval.yaml --target pi-azure

Expected: Score 1.000, assistant content contains the model's response
Actual: Score 0.000, assistant content is empty

Root Cause Analysis

Running the pi CLI directly confirms the model responds correctly:

AZURE_OPENAI_RESOURCE_NAME=wtg-shared-resource \
AZURE_OPENAI_API_KEY=<key> \
pi --provider azure-openai-responses --model gpt-5.4-mini --print --no-session "What is 2+2?"
# Output: 4

Running with --mode json shows the JSONL has content in message_update text deltas, but the final agent_end message has empty assistant content:

{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"2 + 2 = 4"}],...}}
{"type":"agent_end","messages":[{"role":"user",...},{"role":"assistant","content":[],...}]}

Note: message_end HAS content, but agent_end.messages[].content is EMPTY for the assistant message. This is the pi-coding-agent's JSONL serialization issue — it doesn't copy the accumulated content into the agent_end messages array.

What agentv does

The extractMessages function in pi-cli.ts reads from agent_end.messages (primary) or turn_end.message (fallback). A fix was added in this PR to also fall back to message_end events with non-empty content, but the issue is that the message_end content seen in manual runs may not appear when spawned as a subprocess (stdout capture differences on Windows).

Environment

What works

  • subprovider: azure target resolution and alias mapping
  • AZURE_OPENAI_RESOURCE_NAME extraction from base_url
  • pi CLI --provider azure-openai-responses connection to Azure
  • Model returns correct responses when run interactively
  • Azure LLM target (non-agent) scores 1.000 on same evals

What doesn't work

  • pi-cli JSONL agent_end messages have empty assistant content for azure-openai-responses
  • This may be a pi-coding-agent serialization bug or a Windows-specific streaming issue

Files to investigate

  • packages/core/src/evaluation/providers/pi-cli.tsextractMessages(), convertPiMessage()
  • Upstream: @mariozechner/pi-coding-agent JSONL output in --mode json

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions