Codex Backend Architectural Immunity — Capability Consumption, Symmetric Env, and NotImplementedError Arch Tests

[Trecek]

Summary

Add 3 new arch/server tests (plus verify 1 existing) to enforce structural invariants that prevent recurring Codex backend bugs. The tests cover: (1) verifying existing BackendCapabilities consumption test coverage, (2) symmetric env var contract between build_skill_session_cmd and build_food_truck_cmd, (3) no raise NotImplementedError in registered backend classes, and (4) tool-list visibility delta after kitchen enable within a live client session. Also update tests/arch/CLAUDE.md to register the new test files.

Requirements

1. BackendCapabilities Consumption Contract

• Purpose: Every field on BackendCapabilities must be consumed by at least one production read site in src/, OR registered in _FORWARD_DECLARED with a linked issue number
• Location: tests/arch/test_capability_consumption.py — already exists. Verify coverage includes any new fields added by Rectify: Codex MCP Tool/Resource Visibility — notifications/tools/list_changed Immunity #3382 (supports_tool_list_changed). Currently has 4 entries in _FORWARD_DECLARED.
• Note: The _FORWARD_DECLARED dict requires a real #NNNN issue reference per field — enforced by test_forward_declared_has_linked_issues().

2. Symmetric Command Builder Env Vars

• Purpose: build_skill_session_cmd and build_food_truck_cmd on every backend must set the same required base env vars
• Location: tests/arch/ (new test file or extend test_backend_coherence.py)
• Rationale: G5 was caused by the two methods being written independently. build_skill_session_cmd sets AGENT_BACKEND_ENV_VAR, AUTOSKILLIT_HEADLESS_AUTO_GATE, AUTOSKILLIT_APPLICABLE_GUARDS — all missing from build_food_truck_cmd.
• Implementation: Call both methods on each backend in BACKEND_REGISTRY. The food_truck_cmd requires plugin_source arg — use DirectInstall(plugin_dir=Path("/plugins")) per existing pattern.
• Important: Some env vars are intentionally asymmetric. AUTOSKILLIT_SKILL_NAME is IL-1-only (should NOT be in food truck). The test must define a precise "required-in-both" subset: AUTOSKILLIT_HEADLESS, AUTOSKILLIT_SESSION_TYPE, MAX_MCP_OUTPUT_TOKENS, AGENT_BACKEND_ENV_VAR.

3. No NotImplementedError in Production Backends

• Purpose: No method on any backend class registered in BACKEND_REGISTRY should raise NotImplementedError
• Location: tests/arch/ (new test file)
• Rationale: CodingAgentBackend is a typing.Protocol (with @runtime_checkable), NOT an ABC — so there are no legitimate abstract stubs. Any raise NotImplementedError in a registered backend class is a bug.
• Implementation: AST-scan for raise NotImplementedError in classes from BACKEND_REGISTRY.values().
• Note: This test is expected to fail until Rectify: CodexBackend Command Builder Gaps — Missing Env Vars, Sandbox Flags, and Stub Methods #3383 lands (which implements the actual fixes for validate_skill_content and list_plugins).

4. Notification-Support Integration Test

• Purpose: Verify that after open_kitchen is called, the tool list visible to the client actually changes
• Location: tests/server/ (extend test_server_init_gate.py)
• Rationale: G1 was completely invisible to the existing test suite.
• Implementation: Use from fastmcp.client import Client; async with Client(mcp) as client: tools = await client.list_tools() pattern.

Implementation Plan

Plan file: /home/talon/projects/autoskillit-runs/impl-20260530-215854-975848/.autoskillit/temp/make-plan/codex_backend_arch_immunity_plan_2026-05-30_220600.md

🤖 Generated with Claude Code via AutoSkillit

[Trecek]

AutoSkillit PR Review — Verdict: changes_requested

[Trecek]

AutoSkillit review: 1 critical issue (tab/space IndentationError in test_server_init_gate.py) and 2 warnings found. See inline comments.

[Trecek]

Observations accumulated from 1 local review rounds:

[Trecek]

Observation from local review round 0:

[warning] defense: inspect.getsource(cls) can raise OSError (source unavailable, C extension, etc.) with no error context identifying which backend triggered it. Wrap in try/except and include name in the error to preserve context.

Evidence: Marginal improvement in test diagnostics for arch test code. Human judgment needed on whether defensive wrapping in test code is warranted.

[Trecek]

Valid observation — flagged for design decision. The question is whether defensive try/except with backend name context is warranted in test code for inspect.getsource(). Human judgment needed on complexity vs. diagnostic value tradeoff.

[Trecek]

Observation from local review round 0:

[warning] defense: ast.parse(source) can raise SyntaxError with no indication of which backend's source failed to parse. Wrap in try/except and include name in the error message.

Evidence: SyntaxError in well-maintained Python backends is unlikely but possible. Adding try/except with name improves triage. Human decision on whether worth adding complexity to test code.

[Trecek]

Valid observation — flagged for design decision. SyntaxError wrapping in ast.parse() for well-maintained Python backends is unlikely but possible. Human decision on whether the added complexity is worth the diagnostic improvement.