Skip to content

Fix integer overflow in getArraySizeProduct() / ArraySizeProduct() / InnerArraySizeProduct()#107

Open
0xASTRA wants to merge 1 commit into
google:mainfrom
0xASTRA:fix/angle-arraysizeproduct-intof
Open

Fix integer overflow in getArraySizeProduct() / ArraySizeProduct() / InnerArraySizeProduct()#107
0xASTRA wants to merge 1 commit into
google:mainfrom
0xASTRA:fix/angle-arraysizeproduct-intof

Conversation

@0xASTRA
Copy link
Copy Markdown

@0xASTRA 0xASTRA commented May 29, 2026

Summary

TType::getArraySizeProduct() (src/compiler/translator/Types.cpp:560),
gl::ArraySizeProduct() and gl::InnerArraySizeProduct()
(src/common/utilities.cpp:977/987) multiply per-dimension sizes as plain
unsigned int with no overflow handling. A shader declaring
float x[65536][65536] produces a product of 2^32 ≡ 0, which collapses
CalculateVariableSize() to zero and silently passes
TParseContext::checkVariableSize() — the guard (ParseContext.cpp:1767)
that exists specifically to prevent driver-side integer-size overflows in
WebGL contexts (rejectWebglShadersWithLargeVariables = true).

The same wrapped value bypasses the register-packing limit in
VariablePacker.cpp:220 and feeds buffer/loop-count calculations in the
D3D, Metal and HLSL back-ends (~30 call sites, e.g.
ProgramExecutableD3D.cpp:1476).

This is a variant of the class hardened in TIntermAggregate::getConstantValue()
(IntermNode.cpp:879, CheckedNumeric … ValueOrDie(),
crbug.com/498400132) — that fix covered the constant-folding allocation
path but missed these type-level product functions that feed the oversize guard.

Fix

Saturate the product to UINT_MAX on overflow in all three functions,
matching the existing behaviour of getObjectSize() and getLocationCount()
on the same type.

Minimal reproduction

#version 310 es
precision highp float;
float big[65536][65536]; // product 2^32 → wraps to 0
out vec4 col;
void main() { big[0][0] = 1.0; col = vec4(big[0][0]); }
const gl = canvas.getContext('webgl2');
const s = gl.createShader(gl.FRAGMENT_SHADER);
gl.shaderSource(s, src);
gl.compileShader(s);
// shader compiles without error; oversize-variable guard does not fire

Verified: getArraySizeProduct({65536u, 65536u}) returns 0; after fix returns UINT_MAX.

@google-cla
Copy link
Copy Markdown

google-cla Bot commented May 29, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@0xASTRA 0xASTRA force-pushed the fix/angle-arraysizeproduct-intof branch from 786221b to 062e7f3 Compare May 29, 2026 20:46
…ass)

TType::getArraySizeProduct() and gl::ArraySizeProduct()/InnerArraySizeProduct()
multiplied attacker-controlled per-dimension array sizes as plain unsigned int
with no overflow handling. A WebGL shader declaring an array whose dimension
product exceeds UINT_MAX (e.g. float x[65536][65536], product = 2^32 ≡ 0)
caused these functions to return a wrapped, near-zero value.

That wrapped value is consumed by security-relevant arithmetic, notably
CalculateVariableSize() which feeds TParseContext::checkVariableSize() — the
WebGL oversize-variable guard (rejectWebglShadersWithLargeVariables). The
CheckedNumeric there cannot detect that the unsigned int operand already
wrapped, so the oversize variable passes the guard. VariablePacker's packing
limit check is bypassed the same way.

Saturate to UINT_MAX on overflow, mirroring the existing saturating behavior of
TType::getObjectSize() and getLocationCount(), and the CheckedNumeric hardening
recently added to TIntermAggregate::getConstantValue() (crbug.com/498400132).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant