Add the EAGLE-3 speculator CUDA export

[digantdesai]

Exports a registered target (--target-model; gemma4-31B is the reference, see
target.py) and an EAGLE-3 draft head to a single .pte with three methods lowered
together via the CUDA (AOTI) backend: prefill and target_verify (sharing the
target KV cache) and draft_decode (the draft KV cache). The load and the target's
forward-length bounds (min/max tokens in one forward) come from its TargetSpec,
not gemma-specific code; a standalone target decode is omitted because the
shifted runner reseeds from target_verify's feature.

The model stays on the host during export so AOTInductor streams weights to the
GPU per kernel, keeping peak device memory low for the INT4 31B target. For the
static chain_len+1 verify forward to use the small-M INT4 GEMM, the export raises
int4_dispatch.MATVEC_MAX_M to 8 for this export only and restores it afterward;
prefill's dynamic length is then exported with min = max(min_forward_len, 9) and
published as get_min_prefill_chunk (the runner enforces it; no chunking). The
verify window (chain_len+1) is validated against the target's min/max forward
length and the GEMM limit.

Also slices instead of chunks in the draft's rotary helper: chunk lowers to
aten::split_copy, which the AOTI CUDA backend has no fallback kernel for.

Authored with assistance from Claude Code.

[digantdesai]

Stack from ghstack (oldest at bottom):

• Update tokenizers submodule to the linear-time HF encode path #20349
• Add --prompt_file flag to the eagle3 runner #20348
• Add chunked prefill to the eagle3 runner for prompts past the ring-cache cap #20347
• Fix sliding-window chunked prefill in the gemma4-31B runner #20346
• Make EAGLE-3 chain length K a runtime parameter #20345
• Add split-K mid-M SDPA Triton kernel for EAGLE-3 target_verify at long context #20344
• Add EAGLE-3 end-to-end speculative-decode test #20157
• Add the EAGLE-3 speculative-decoding runner (CUDA) #20156
• -> Add the EAGLE-3 speculator CUDA export #20155
• Add a weight-stationary INT4 GEMM for small M #20154
• Add the Eagle3Speculator module for speculative decoding #20153
• Add KV cache to the EAGLE-3 draft head #20152
• Add Gemma4 EAGLE-3 eager reference #20151
• Add Gemma4 EAGLE-3 hidden-state taps #20150
• Add EAGLE-3 draft head #20149

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20155

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 2 Active SEVs

There are 2 currently active SEVs. If your PR is affected, please view them below:

• [ROCm] MI350 CI runner label rename: rebase PRs using old linux.rocm.gpu.gfx950.* labels
• [Housekeeping] Rename uw1 cluster to meta-prod-aws-uw1

❌ 1 New Failure, 3 Unclassified Failures

As of commit 5099f90 with merge base da9158b ():

NEW FAILURE - The following job has failed:

• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t c47e4e720db80d9a08a44063c52701a0e99f0a14f59d6984e460370aef86202b /exec failed with exit code 137

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Aarch64 Linux Wheels / pytorch/executorch / build-wheel-py3_10-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /__w/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/inmemoryfs/inmemory_filesystem.cpp:722:48: error: ‘inmemoryfs::InMemoryFileSystem::InMemoryNode::Kind’ has not been declared
• Build Aarch64 Linux Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_aarch64
• Test CUDA Builds / unittest-cuda / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_int4_dispatch.py::TestLimitConsistency::test_shim_gemm_max_m_matches_cuh

This comment was automatically generated by Dr. CI and updates every 15 minutes.