Add the EAGLE-3 speculative-decoding runner (CUDA)

[digantdesai]

A C++ runner that drives the speculator .pte with the shifted (vLLM-EAGLE)
scheme: the draft pairs the target hidden state at position t with token t+1, so
each round runs one target forward (target_verify) and reseeds the next draft
chain from the hidden states verify already produced -- no standalone target
decode. Greedy verification keeps output identical to greedy target decoding.
target_verify runs on stable input buffers and can be captured as a CUDA graph.

It requires the .pte metadata (fails loudly if absent) and enforces the exported
prefill range [get_min_prefill_chunk, get_max_prefill_chunk] (no chunking). The
prefill bonus token is always emitted; the speculative loop runs only when more
tokens are requested, the bonus was not EOS, and a K-token verify window fits
within get_max_seq_len (so a one-token or near-context request returns without
seeding the draft). The chat template and stop tokens are flags defaulting to
Gemma 4 IT (--chat_prefix/--chat_suffix/--stop_ids/--stop_token, --bos_id -1 to
skip) so other target/tokenizer pairs run without code changes. Device-to-host
reads are error-checked; the printed tau excludes the free prefill token.

Authored with assistance from Claude Code.

[digantdesai]

Stack from ghstack (oldest at bottom):

• Update tokenizers submodule to the linear-time HF encode path #20349
• Add --prompt_file flag to the eagle3 runner #20348
• Add chunked prefill to the eagle3 runner for prompts past the ring-cache cap #20347
• Fix sliding-window chunked prefill in the gemma4-31B runner #20346
• Make EAGLE-3 chain length K a runtime parameter #20345
• Add split-K mid-M SDPA Triton kernel for EAGLE-3 target_verify at long context #20344
• Add EAGLE-3 end-to-end speculative-decode test #20157
• -> Add the EAGLE-3 speculative-decoding runner (CUDA) #20156
• Add the EAGLE-3 speculator CUDA export #20155
• Add a weight-stationary INT4 GEMM for small M #20154
• Add the Eagle3Speculator module for speculative decoding #20153
• Add KV cache to the EAGLE-3 draft head #20152
• Add Gemma4 EAGLE-3 eager reference #20151
• Add Gemma4 EAGLE-3 hidden-state taps #20150
• Add EAGLE-3 draft head #20149

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20156

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 2 Active SEVs

There are 2 currently active SEVs. If your PR is affected, please view them below:

• [ROCm] MI350 CI runner label rename: rebase PRs using old linux.rocm.gpu.gfx950.* labels
• [Housekeeping] Rename uw1 cluster to meta-prod-aws-uw1

❌ 1 New Failure, 1 Unrelated Failure, 2 Unclassified Failures

As of commit 593f812 with merge base da9158b ():

NEW FAILURE - The following job has failed:

• Lint / lintrunner (gh)
  >>> Lint for examples/models/eagle3/main.cpp:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Aarch64 Linux Wheels / pytorch/executorch / build-wheel-py3_10-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /__w/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/inmemoryfs/inmemory_filesystem.cpp:722:48: error: ‘inmemoryfs::InMemoryFileSystem::InMemoryNode::Kind’ has not been declared
• Build Aarch64 Linux Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_aarch64

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.