-
Notifications
You must be signed in to change notification settings - Fork 0
pi-cli azure: empty assistant content in agent_end JSONL #922
Description
Summary
When using the pi-cli provider with subprovider: azure and azure-openai-responses, the pi CLI returns empty assistant content in the agent_end JSONL event. This causes all pi-cli azure evals to score 0.
Context
PR #921 added azure subprovider support for pi-coding-agent and pi-cli providers. The agentv connectivity, auth, and target resolution all work correctly. The pi CLI successfully connects to Azure, sends prompts, and the model generates responses. However, the response text is not populated in the JSONL output.
Reproduction
# targets.yaml
- name: pi-azure
provider: pi-cli
subprovider: azure
base_url: ${{ AZURE_OPENAI_ENDPOINT }} # resource name or URL
model: ${{ AZURE_DEPLOYMENT_NAME }}
api_key: ${{ AZURE_OPENAI_API_KEY }}
grader_target: grader# Run a simple eval
GRADER_TARGET=azure agentv eval run evals/self/azure-smoke.eval.yaml --target pi-azureExpected: Score 1.000, assistant content contains the model's response
Actual: Score 0.000, assistant content is empty
Root Cause Analysis
Running the pi CLI directly confirms the model responds correctly:
AZURE_OPENAI_RESOURCE_NAME=wtg-shared-resource \
AZURE_OPENAI_API_KEY=<key> \
pi --provider azure-openai-responses --model gpt-5.4-mini --print --no-session "What is 2+2?"
# Output: 4Running with --mode json shows the JSONL has content in message_update text deltas, but the final agent_end message has empty assistant content:
{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"2 + 2 = 4"}],...}}
{"type":"agent_end","messages":[{"role":"user",...},{"role":"assistant","content":[],...}]}Note: message_end HAS content, but agent_end.messages[].content is EMPTY for the assistant message. This is the pi-coding-agent's JSONL serialization issue — it doesn't copy the accumulated content into the agent_end messages array.
What agentv does
The extractMessages function in pi-cli.ts reads from agent_end.messages (primary) or turn_end.message (fallback). A fix was added in this PR to also fall back to message_end events with non-empty content, but the issue is that the message_end content seen in manual runs may not appear when spawned as a subprocess (stdout capture differences on Windows).
Environment
- Windows 11
- Node.js v22.22.1
- pi-coding-agent v0.64.0 (npm global)
- agentv v4.6.0 (local build from PR feat(pi): add azure subprovider support with base_url config #921)
What works
subprovider: azuretarget resolution and alias mappingAZURE_OPENAI_RESOURCE_NAMEextraction from base_url- pi CLI
--provider azure-openai-responsesconnection to Azure - Model returns correct responses when run interactively
- Azure LLM target (non-agent) scores 1.000 on same evals
What doesn't work
- pi-cli JSONL
agent_endmessages have empty assistant content for azure-openai-responses - This may be a pi-coding-agent serialization bug or a Windows-specific streaming issue
Files to investigate
packages/core/src/evaluation/providers/pi-cli.ts—extractMessages(),convertPiMessage()- Upstream:
@mariozechner/pi-coding-agentJSONL output in--mode json