Skip to content

Disable qnn_16a16w Llama runner test (OOM on linux.2xlarge)#20511

Merged
psiddh merged 1 commit into
pytorch:mainfrom
psiddh:disable-qnn-16a16w-oom
Jun 25, 2026
Merged

Disable qnn_16a16w Llama runner test (OOM on linux.2xlarge)#20511
psiddh merged 1 commit into
pytorch:mainfrom
psiddh:disable-qnn-16a16w-oom

Conversation

@psiddh

@psiddh psiddh commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

The test-llama-runner-qnn-linux (qnn_16a16w) job has been OOM-killed on linux.2xlarge since PR #19660 landed, blocking viable/strict from advancing for 73+ commits. Disable it while the Qualcomm team investigates the memory regression and potential accuracy issue.

The test-llama-runner-qnn-linux (qnn_16a16w) job has been OOM-killed on
linux.2xlarge since PR pytorch#19660 landed, blocking viable/strict from
advancing for 73+ commits. Disable it while the Qualcomm team
investigates the memory regression and potential accuracy issue.
Copilot AI review requested due to automatic review settings June 25, 2026 06:44
@pytorch-bot

pytorch-bot Bot commented Jun 25, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20511

Note: Links to docs will display an error until the docs builds have been completed.

❗ 2 Active SEVs

There are 2 currently active SEVs. If your PR is affected, please view them below:

❌ 3 New Failures, 3 Unrelated Failures, 1 Unclassified Failure

As of commit e620e96 with merge base aada6d7 (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 25, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR temporarily removes the qnn_16a16w configuration from the Llama runner QNN Linux CI matrix to prevent OOM failures on linux.2xlarge, unblocking viable/strict while the underlying regression is investigated.

Changes:

  • Drop qnn_16a16w from the pt2e_quantize matrix for the test-llama-runner-qnn-linux job in both trunk and PR workflows.
  • Add a TODO note indicating the configuration should be re-enabled once the OOM issue is resolved.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/trunk.yml Removes qnn_16a16w from the trunk CI matrix for the QNN Llama runner job.
.github/workflows/pull.yml Removes qnn_16a16w from the PR CI matrix for the QNN Llama runner job.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +954 to +955
# TODO(T12345): re-enable qnn_16a16w once OOM on linux.2xlarge is resolved
pt2e_quantize: [qnn_8a8w]
Comment on lines +828 to +829
# TODO(T12345): re-enable qnn_16a16w once OOM on linux.2xlarge is resolved
pt2e_quantize: [qnn_8a8w]
@psiddh

psiddh commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

@winskuo-quic can you pls approve it if [test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job ) passes on this PR ?

@winskuo-quic winskuo-quic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks a lot for the support

@larryliu0820 larryliu0820 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tried linux.4xlarge?

@psiddh psiddh merged commit a5f9893 into pytorch:main Jun 25, 2026
180 of 193 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants