[aw-failures] [fix][P0] Pin Codex CLI back from 0.133.0 — `stream_options.include_usage` rejected by `gpt-5.5`

[github-actions]

Problem statement

After PR #34390 (merged 2026-05-24T12:59:44 UTC) bumped DefaultCodexVersion from 0.130.0 to 0.133.0, every Codex-engine workflow run fails deterministically with HTTP 400 Unknown parameter: 'stream_options.include_usage' from the OpenAI chat completions API. The Codex CLI retry budget (3 retries on top of the initial attempt = 4 attempts) exhausts on identical errors and the run exits with code 1.

This breaks all Codex-engine workflows on main. See the parent report #34521 for the full 6h failure analysis.

Affected workflows and run IDs

Last 6h (2026-05-24 17:56 → 19:10 UTC):

Workflow	Run	Event
Daily Cache Strategy Analyzer	§26369636900	schedule
Changeset Generator	§26369303262	pull_request
Smoke Codex	§26369303293	pull_request
Changeset Generator	§26368932369	pull_request
Smoke Codex	§26368932442	pull_request
Changeset Generator	§26368599354	pull_request
Smoke Codex	§26368599382	pull_request

All 7 runs use engine_id: codex, model: gpt-5.5, codex_app_server.client_version: 0.133.0.

Probable root cause

Codex CLI 0.133.0 sends stream_options.include_usage: true in the chat completions request body. The gpt-5.5 model endpoint rejects this parameter:

{
  "error": {
    "message": "Unknown parameter: 'stream_options.include_usage'.",
    "type": "invalid_request_error",
    "param": "stream_options.include_usage",
    "code": "unknown_parameter"
  }
}

This is an upstream Codex regression — the CLI emits an OpenAI SDK parameter that the configured GitHub-models-routed gpt-5.5 endpoint does not accept. The previous pin (0.130.0) does not emit this field and has been observed working in baseline runs (e.g. §26184060675, Daily Cache Strategy Analyzer, 33 OpenAI requests, conclusion success).

Specific proposed remediation

Pick one (recommended first):

1. Partial revert of Bump pinned Copilot/Codex/GitHub MCP versions and regenerate workflow artifacts #34390: open a follow-up PR that pins DefaultCodexVersion back to 0.130.0 while keeping DefaultCopilotVersion: 1.0.52 and DefaultGitHubMCPVersion: v1.0.5. Regenerate lockfiles. Constants live in:
   • pkg/workflow/codex_engine.go (search for DefaultCodexVersion)
   • update pkg/workflow/js/version.js and any per-workflow lockfiles regenerated by make regen.
2. Wait for Codex 0.134+: if upstream ships a fix that omits stream_options.include_usage when the target endpoint doesn't accept it, bump to the patched version instead of reverting.
3. Upstream report: file a bug at openai/codex (or the appropriate GitHub-hosted equivalent) attaching the request trace and the gpt-5.5 400 response. Note that codex_app_server.client_version: 0.133.0 is the breaking version.

Success criteria / verification

• DefaultCodexVersion no longer 0.133.0 on main.
• Smoke Codex run on a fresh PR succeeds (conclusion: success, error_count: 0).
• Daily Cache Strategy Analyzer next scheduled run completes with at least 1 successful agent turn (compare against baseline of 33 OpenAI requests in audit-diff).
• No new occurrences of stream_options.include_usage in agent-stdio logs across the 6h window after the fix lands.

Parent report

#34521
Related to #34521

Generated by 🔍 [aw] Failure Investigator (6h) · opus47 15.3M · ◷

• expires on May 31, 2026, 7:22 PM UTC

---

Problem statement

After PR #34390 (merged 2026-05-24T12:59:44 UTC) bumped DefaultCodexVersion from 0.130.0 to 0.133.0, every Codex-engine workflow run fails deterministically with HTTP 400 Unknown parameter: 'stream_options.include_usage' from the OpenAI chat completions API. The Codex CLI retry budget (3 retries on top of the initial attempt = 4 attempts) exhausts on identical errors and the run exits with code 1.

This breaks all Codex-engine workflows on main, regardless of model. See the parent report #34521 for the original 6h failure analysis.

Scope broader than gpt-5.5 — gpt-5.4 also affected

The original report focused on gpt-5.5-targeted workflows. Re-investigation on 2026-05-25 confirms the same error on gpt-5.4. The faulty parameter is emitted by Codex CLI 0.133.0 client-side; the rejecting endpoint behavior is shared by the GitHub-models routed gpt-5.4 and gpt-5.5 endpoints.

Updated affected runs (added 2026-05-25 investigation)

Daily Observability Report for AWF Firewall and MCP Gateway (engine: codex, model: gpt-5.4) has failed 5 of the last 8 daily scheduled runs:

Date	Run	Conclusion
2026-05-25	§26376474020	failure (same stream_options.include_usage error confirmed in job log line 7941)
2026-05-24	§26346978327	failure
2026-05-23	§26317790418	failure
2026-05-22	§26260337852	failure
2026-05-21	§26197188358	success
2026-05-20	§26132834933	failure

Last 6h (2026-05-24 17:56 → 2026-05-25 01:30 UTC) Codex-engine failures still observed:

Workflow	Run	Engine / Model
Daily Observability Report for AWF Firewall and MCP Gateway	§26376474020	codex / gpt-5.4

Probable root cause (unchanged)

Codex CLI 0.133.0 sends stream_options.include_usage: true in the chat completions request body. The endpoint rejects this parameter:

{
  "error": {
    "message": "Unknown parameter: 'stream_options.include_usage'.",
    "type": "invalid_request_error",
    "param": "stream_options.include_usage",
    "code": "unknown_parameter"
  }
}

This is an upstream Codex regression — the CLI emits an OpenAI SDK parameter that the configured GitHub-models-routed endpoints (both gpt-5.4 and gpt-5.5) do not accept. The previous pin (0.130.0) does not emit this field and has been observed working in baseline runs.

Current state of fix

As of 2026-05-25T01:30 UTC, pkg/constants/version_constants.go still has:

const DefaultCodexVersion Version = "0.133.0"

No revert PR has been merged. The bug remains live in production.

Specific proposed remediation (unchanged)

Pick one (recommended first):

1. Partial revert of Bump pinned Copilot/Codex/GitHub MCP versions and regenerate workflow artifacts #34390: open a follow-up PR that pins DefaultCodexVersion back to 0.130.0 while keeping DefaultCopilotVersion: 1.0.52 and DefaultGitHubMCPVersion: v1.0.5. Regenerate lockfiles. Constants live in:
   • pkg/constants/version_constants.go (DefaultCodexVersion)
   • regenerate any per-workflow lockfiles via make regen.
2. Wait for Codex 0.134+: if upstream ships a fix that omits stream_options.include_usage when the target endpoint doesn't accept it, bump to the patched version instead of reverting.

Success criteria

• DefaultCodexVersion no longer emits stream_options.include_usage against GitHub-models gpt-5.4 / gpt-5.5 endpoints.
• Daily Observability Report and other Codex workflows produce successful runs for 3 consecutive scheduled executions.
• No new occurrences of HTTP 400 unknown_parameter for stream_options.include_usage in the Codex CLI retry logs.

---

_{Updated 2026-05-25 by failure-investigator with new gpt-5.4 evidence. Lookback: 6h. Triaged failures: 3 (1 Codex covered here; 2 transient Claude one-offs not requiring tracking).}

Generated by 🔍 [aw] Failure Investigator (6h) · opus47 19.7M · ◷