Qualcomm AI Engine Direct - AMD backend error

[winskuo-quic]

Summary

We noticed that when performing inference with AMD CPU, we will run into Floating point exception (core dumped).
This can be easily reproduced with following lines of code:

import torch.nn as nn
import torch
w2_conv = nn.Conv2d(1536, 32, 1, bias=False)
x = torch.randn(1,1536,1,32)
w2_conv(x)

Temp solution is to set mkldnn.enabled=False:
torch.backends.mkldnn.enabled = False

Test plan

NA

cc @cccclai @cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18098

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures

As of commit 6455008 with merge base fde943a ():

NEW FAILURES - The following jobs have failed:

• Build Cadence / cpu-x86 / build (gh)
  RuntimeError: Command docker exec -t 561b89c281af7daa963b6ca6f2ff4ef49ae8e65e1dc40db441996809c5afd9cc /exec failed with exit code 1
• Lint / link-check / lint-urls (gh)
  The process '/usr/bin/git' failed with exit code 128
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-models-linux (mobilebert, portable, linux.2xlarge) / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-models-linux (resnet18, portable, linux.2xlarge) / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-models-linux-basic (vit, portable, cmake, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11... / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-moshi-linux / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t d586f4319299e540e4c0f685fff01976ef74ce4cc9705353414b30a2efc1b06a /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t dae44f75b3580ac143cd8fb7d0ab7a0ef020bc3e16de74e67b07c6c56e1bc730 /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t 3efdaace73dc80f6a7da85cbab630bdbb830ddaa39df0c7947e37f0a2b6b5f26 /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.13) / linux-job (gh)
  RuntimeError: Command docker exec -t 6b18830f3ef8bc97e160725cb1b4593730b79d2dff811a59a62eab75338aa01b /exec failed with exit code 1
• pull / test-selective-build-linux / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / unittest-editable / windows / windows-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).
• Test CUDA Builds / check-all-cuda-builds (gh)
  Process completed with exit code 1.
• Test CUDA Builds / test-executorch-cuda-build-13.0 / linux-job (gh)
  ##[error]Response status code does not indicate success: 401 (Unauthorized).

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[winskuo-quic]

Hi @cccclai, @abhinaykukkadapu,
We have noticed that AMD CPU during AOT will run into the error: Floating point exception (core dumped). This happens during inference, including nn.Module.
There's a sample in summary section to reproduce.
This PR is a quick workaround to fix the issue, but I am assuming if this is a AMD or Torch issue, placing these logic under QNN probably isn't the best option.
Please have a look.
Thanks

[digantdesai]

I assume this is during eager model runs?

[digantdesai]

Would you mind creating a ticket on PyTorch/PyTorch?

[winskuo-quic]

I assume this is during eager model runs?

Hi @digantdesai,
This issue happened in both Eager Model and Exported Program when we are calibrating the model.
I have created an issue ticket under pytorch/pytorch: pytorch/pytorch#177227