fix(oom): multimodal OOM fix with auto-profile

[sufubao]

Summary

Replaces PR #1253 with a complete multimodal OOM fix covering:

• Layer 0: min → max bugfix at visualserver/manager.py:87 that was silently capping per-DP visual batch size to 1.
• Layer 1: per-model stress helpers for Qwen2_VL, Qwen2_5_VL, Qwen3_VL, Qwen3_omni_moe that hold the stress peak at the driver level by deliberately omitting torch.cuda.empty_cache(). Visual and audio runtime semaphores tightened from × 8 to × 1 so concurrency cannot exceed the stress-tested peak.
• Layer 2: LLM-side two-pass auto-profile that eliminates --mem_fraction tuning. Phase 1 runs a probe-sized KV + real prefill + real decode stress. Phase 2 computes the target KV size from measured peak. Phase 3 tears down + empty_caches + reallocs at target + re-captures graphs + allocates a 256 MB canary. Phase 4 validates. On OOM, shrinks 5% and retries (cap 3 retries, 4 total attempts) then fails fast with a multi-knob diagnostic.
• --mem_fraction default flipped from 0.9 to 1.0 (now an optional paranoia multiplier).
• DISABLE_CHECK_MAX_LEN_INFER env var removed — use --max_total_token_num to pin a size instead.

See docs/superpowers/specs/2026-04-16-multimodal-oom-fix-design.md for the full design.

Test plan

• pure-Python unit tests in unit_tests/common/basemodel/test_auto_profile.py pass (8/8)
• local GPU smoke test on Qwen2.5-VL-7B — auto-profile happy path
• local GPU smoke test — explicit --max_total_token_num escape hatch
• local GPU smoke test — forced failure path raises with diagnostic
• 1-hour replay on staging replica of the production Qwen3.5-VL workload, all acceptance criteria from spec §10.3 green

Notes

Do not merge to origin/main. This PR is for review only pending maintainer sign-off after the staging replay validates the four-commit stack.

[gemini-code-assist]

Code Review

This pull request implements a two-pass auto-profiling mechanism and a "reserve-then-yield" memory contract to prevent multimodal CUDA OOM errors. It updates subprocess initialization, adds realistic stress tests for vision models, and tightens concurrency limits. Feedback indicates that the Phase 3 rebuild loop is missing necessary re-initialization calls for the attention backend and padded requests, and the teardown process fails to clear references to the attention backend and OOM canary, which could prevent memory from being released.

[gemini-code-assist]

The rebuild loop in Phase 3 is missing calls to self._init_att_backend() and self._init_padded_req(). Since mem_manager and req_manager are recreated, the attention backend and the padded request (which hold references to the old managers or their internal state like KV buffer pointers and request slots) must also be re-initialized. Without this, the model will continue to use stale state pointing to the released probe memory, leading to crashes or incorrect results during validation and service.

Suggested change

	# Phase 3: re-init everything at target_tokens
	self.max_total_token_num = target_tokens
	self._init_mem_manager()
	self._init_kv_move_buffer()
	self._check_mem_size()
	self._init_req_manager()
	self._init_cudagraph()
	self._init_prefill_cuda_graph()
	# Phase 3: re-init everything at target_tokens
	self.max_total_token_num = target_tokens
	self._init_mem_manager()
	self._init_kv_move_buffer()
	self._check_mem_size()
	self._init_req_manager()
	self._init_att_backend()
	if hasattr(self, "_init_att_backend1"):
	self._init_att_backend1()
	self._init_padded_req()
	self._init_cudagraph()
	self._init_prefill_cuda_graph()

[gemini-code-assist]

The _teardown_graphs_and_kv method is missing the cleanup of self.att_backend (and self.att_backend1 if present) and self._oom_canary. If these references are not dropped, torch.cuda.empty_cache() will not be able to release the memory occupied by the probe KV buffer and the canary tensor, which will cause the teardown leak check to fail or lead to OOM during the rebuild phase.

    def _teardown_graphs_and_kv(self):
        """Phase 3 teardown: drop references to the probe's kv_buffer and
        all captured CUDA graphs so torch.cuda.empty_cache() can actually
        return the blocks to the driver.

        Order matters — graphs hold tensor pointers into kv_buffer and must
        go first, then req_manager (which references mem_manager), then
        mem_manager itself. See spec §6.5 for the rationale.
        """
        if hasattr(self, "mem_manager") and self.mem_manager is not None:
            try:
                self.mem_manager.free_all()
            except Exception:
                pass

        for attr in ("graph", "prefill_graph", "prefill_cuda_graph", "graph1", "prefill_graph1"):
            if hasattr(self, attr):
                setattr(self, attr, None)

        if hasattr(self, "att_backend"):
            self.att_backend = None
        if hasattr(self, "att_backend1"):
            self.att_backend1 = None
        if hasattr(self, "_oom_canary"):
            self._oom_canary = None

        if hasattr(self, "req_manager"):
            self.req_manager = None
        if hasattr(self, "mem_manager"):
            self.mem_manager = None