Fix 32-bit integer overflow in LAPACKE nancheck index arithmetic#5835
Merged
martin-frbg merged 1 commit intoJun 12, 2026
Merged
Conversation
The optional NaN checks performed by LAPACKE front-ends computed array offsets and lengths in 32-bit lapack_int arithmetic, which overflows for large matrices (e.g. LAPACKE_dpotrf with n = lda >= 46341 overflows j*lda in LAPACKE_dtr_nancheck, leading to out-of-bounds reads and crashes). Several nancheck helpers (ge, gb, tp, tf) already cast to size_t; this applies the same treatment to the ones that were missed: - ?tr_nancheck: cast a[i+j*lda] index to size_t (also covers po/sy/he checks, i.e. the POTRF/Cholesky path) - ?_nancheck (vector): iterate by element count instead of n*inc, with size_t indexing - ?sp/?pp/?pf/?hp/?tp/?tf_nancheck: compute n*(n+1)/2 length in size_t - ?tz_nancheck: widen the part offsets to int64_t (keeping the -1 sentinel) and compute them with a widening cast Verified with UBSan: LAPACKE_str_nancheck at n = lda = 47000 previously reported signed integer overflow and crashed with SEGV at -O0; it now passes and still detects the NaN. Old and new code produce identical results across 1907 enumerated small-size cases (all layouts, uplo, diag, shapes and NaN positions for tr/tz/vector/packed variants). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
changangela
force-pushed
the
fix-lapacke-nancheck-overflow
branch
from
7f7585d to
ede136e
Compare
Collaborator
|
Thank you - and yes please submit this fix to Reference-LAPACK as well, we're only copying LAPACKE and (most of) LAPACK from there |
Contributor
Author
sounds good, here it is: Reference-LAPACK/lapack#1294 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The LAPACKE NaN checks compute array offsets in 32-bit
lapack_int, which overflows for large matrices. Found while running Cholesky on a 47000x47000 matrix:LAPACKE_dpotrfwithn = lda >= 46341crashes inLAPACKE_dtr_nancheckbefore the factorization even runs, becausej*ldawraps negative. Same family of overflow as scipy/scipy#20229.The
ge/gb/tp/tfhelpers already cast tosize_t, so this applies the same fix to the ones that were missed:?tr_nancheck: casta[i+j*lda]tosize_t(covers the po/sy/he checks, i.e. potrf)?_nancheck(vector): then*incloop bound overflowed; iterate by count and index withsize_t?sp/?pp/?pf/?hp/?tp/?tf: compute the packed lengthn*(n+1)/2insize_t?tz_nancheck: widen the part offsets toint64_tTested with UBSan at n = 47000: the old code overflows and segfaults (tr, tz) or silently returns "no NaN" on input that has one (vector, packed); the fixed code detects a NaN planted at the far end of the scanned region in all cases. Old and new code give identical results on an exhaustive sweep of small sizes (all layouts/uplo/diag/shapes, NaN at every position).
Reference-LAPACK master has the same bug in these files; I can submit the fix there as well.