Skip to content

Qualcomm: cap inf replacement value to fix 16a16w accuracy regression #20471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
psiddh wants to merge 3 commits into
base: main
Choose a base branch
from
+6 −1
Open

Qualcomm: cap inf replacement value to fix 16a16w accuracy regression #20471

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2333727
Qualcomm: cap inf replacement value to fix 16a16w accuracy regression
psiddh Jun 24, 2026
45a7414
Merge branch 'main' into fix-qnn-16a16w-inf-replacement
psiddh Jun 24, 2026
87c4feb
Merge branch 'main' into fix-qnn-16a16w-inf-replacement
psiddh Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion backends/qualcomm/quantizer/quantizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,7 +416,12 @@ def _get_quant_range(self, node):
if quant_info.output_qspec.quant_min is None
else quant_info.output_qspec.quant_min
)
return quant_range
# Cap the inf stand-in so it does not dominate the tensor's
# dynamic range. For >8-bit activations the full range (e.g.
# 65535 for uint16) would blow up the attention-mask quant scale
# and wreck accuracy; 255 keeps a reasonable scale for
# Llama-style attention masks.
return min(quant_range, 255)

@winskuo-quic winskuo-quic Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @psiddh,
Thanks for identifying the issue.
Could you please let me do a quick test on the CI testing model and validate the accuracy issue?
Ideally, we don't want hard coded numbers, and that is why we replaced them with quant range in the PR: #19660

@psiddh psiddh Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, please go ahead and validate on CI, happy to wait for your results.

On the hardcoded 255: agree we should avoid magic numbers. perhaps a cleaner version would be to derive the cap from the int8 range rather than hardcoding: ..

return min(quant_range, torch.iinfo(torch.int8).max - torch.iinfo(torch.int8).min)

@psiddh psiddh Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@winskuo-quic Any updates ? We would like to land this asap as this is stalling viable/strict branch from moving forward.

@psiddh psiddh Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@winskuo-quic The CI runner for test-llama-runner-qnn-linux (qnn_16a16w) is OOM-killing the export step, exit code 137 (SIGKILL), not a test accuracy failure. Your fix is correct (the inf cap logic is sound), but it's never getting far enough to prove it, the Python export process gets killed before it finishes.
Some OOM that's been killing it on trunk since #19660 perhaps !


def _get_candidates_with_infinity_args(self, graph_module: GraphModule):
binary_op_sources = [
Expand Down
Loading