Update list.{count, index, remove} to accept object type

[randolf-scholz]

Partially fixes #15271

[randolf-scholz]

Hm, not exactly sure what's going on with numpy, my best guess is that since _NestedSequence is defined with (https://github.com/numpy/numpy/blob/main/numpy/_typing/_nested_sequence.py)

    def count(self, value: Any, /) -> int:
        """Return the number of occurrences of `value`."""
        raise NotImplementedError

    def index(self, value: Any, /) -> int:
        """Return the first index of `value`."""
        raise NotImplementedError

then after we make list use object rather than Any, this is still a subtype but possibly no longer a "proper_subtype". This is a mypy interal distinction and likely needs to be addressed on mypys side.

EDIT: I am starting to think that this may be related to mypy interal solver semantics. we removed some constrains, so now it may find solutions for the T of a list-constructor inside nested function calls, where previously there was no solution and it defaulted to Never which then propagate throught the call stack and may change inference in some cases.

[srittau]

The primer output is not promising. The fixed # type: ignores are mostly in test code. On the other hand there seems to be some churn, and even some true positives (in colour) that aren't flagged anymore.

[randolf-scholz]

The main issue seems to be [var-annotated] issue originating from numpy:

import numpy as np

reveal_type(np.array([]))

# on main
tmp_a.py:3: note: Revealed type is "numpy.ndarray[tuple[Any, ...], numpy.dtype[Any]]"
# PR:
tmp_a.py:3: note: Revealed type is "numpy.ndarray[tuple[Any, ...], numpy.dtype[Never]]"

[randolf-scholz]

@srittau I wouldn't call the colour results true-positives. For instance https://github.com/colour-science/colour/blob/c9cfa1f333452cc05e42f09eebd2f3dd935a5981/colour/utilities/network.py#L1756-L1773 is exactly one of the cases of a valid list.remove call that is currently a false positive.

[randolf-scholz]

OK, I think I have a repro of the numpy issue that causes the [var-annotated] errors. It has to do with the NestedSequence Protocol. Here is a minimal reproduction:

from typing import Any, Protocol, reveal_type

class NestedSeq[_T_co](Protocol):

    def __getitem__(self, index: int, /) -> "_T_co | NestedSeq[_T_co]":
        raise NotImplementedError

    def count(self, value: Any, /) -> int:
        raise NotImplementedError

def as_nested_sequence[T](seq: NestedSeq[T], /) -> NestedSeq[T]:
    return seq

reveal_type(as_nested_sequence([]))
# main: NestedSeq[Any], PR: NestedSeq[Never]

And this happens due to how mypy solves type-vars. on main, count is [T] -> int, so mypy solves T=Any, whereas on PR, count is [object] -> int, so mypy has no constraints for T and picks T=Never.

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

[jorenham]

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

Sure, we could do that.

[jorenham]

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

Sure, we could do that.

The workaround will be present in the upcoming 2.4.5 and 2.5.0 releases.

[jorenham]

numpy 2.4.5 was just released (https://github.com/numpy/numpy/releases/tag/v2.4.5)

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

optuna (https://github.com/optuna/optuna)
+ tests/visualization_tests/test_rank.py:699: error: Need type annotation for "x2"  [var-annotated]

colour (https://github.com/colour-science/colour)
- colour/utilities/network.py:1772: error: Argument 1 to "remove" of "list" has incompatible type "PortNode"; expected "Self"  [arg-type]
- colour/notation/munsell/centore2014.py:2053: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/notation/munsell/centore2014.py:2058: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/notation/munsell/centore2014.py:2065: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/notation/munsell/centore2014.py:2070: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]"  [arg-type]
- colour/plotting/models.py:1457: error: Incompatible types in assignment (expression has type "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
+ colour/plotting/models.py:1457: error: Incompatible types in assignment (expression has type "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", variable has type "ndarray[Any, Any]")  [assignment]

pwndbg (https://github.com/pwndbg/pwndbg)
+ pwndbg/aglib/elf.py:448: error: Unused "type: ignore" comment  [unused-ignore]

dedupe (https://github.com/dedupeio/dedupe)
+ dedupe/labeler.py:410: error: List item 1 has incompatible type "Iterable[Literal[0, 1]]"; expected "_SupportsArray[dtype[signedinteger[_64Bit]]] | _NestedSequence[_SupportsArray[dtype[signedinteger[_64Bit]]]]"  [list-item]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/dtypes/dtypes.py:983: error: Need type annotation for "np_dtype"  [var-annotated]
+ pandas/core/dtypes/dtypes.py:984: error: Argument 1 has incompatible type "list[dtype[Any] | ExtensionDtype]"; expected "_SupportsArray[dtype[Never]] | _NestedSequence[_SupportsArray[dtype[Never]]]"  [arg-type]
+ pandas/core/arrays/sparse/array.py:1526: error: Need type annotation for "sparse_values"  [var-annotated]
+ pandas/core/internals/managers.py:2502: error: List comprehension has incompatible type List[ndarray[tuple[Any, ...], dtype[Any]] | ExtensionArray]; expected List[_SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]]]  [misc]
+ pandas/plotting/_matplotlib/misc.py:170: error: Need type annotation for "s"  [var-annotated]
+ pandas/plotting/_matplotlib/core.py:1207: error: List item 0 has incompatible type "Number | number[Any, int | float | complex]"; expected "_SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]]"  [list-item]

xarray (https://github.com/pydata/xarray)
+ xarray/tests/test_concat.py: note: In function "create_typed_datasets":
+ xarray/tests/test_concat.py:112: error: Argument 1 to "reshape" has incompatible type "list[Timedelta]"; expected "_SupportsArray[dtype[Never]] | _NestedSequence[_SupportsArray[dtype[Never]]]"  [arg-type]
+ xarray/tests/test_concat.py: note: At top level:
+ xarray/tests/test_backends.py: note: In member "test_mask_and_scale" of class "NetCDF4Base":
+ xarray/tests/test_backends.py:2090: error: Need type annotation for "expected"  [var-annotated]
+ xarray/tests/test_backends.py: note: At top level:

scipy-stubs (https://github.com/scipy/scipy-stubs)
+ tests/linalg/test_blas.pyi:27: error: Expression is of type "list[_FortranFunction]", not "list[_FortranFunction] | _FortranFunction"  [assert-type]
+ tests/linalg/test__decomp_svd.pyi:72: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/linalg/test__decomp_svd.pyi:73: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/linalg/test__decomp_svd.pyi:74: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/special/test_ufuncs.pyi:75: error: Argument 1 to "__call__" of "_UFunc11f" has incompatible type "list[complex]"; expected "CanArrayND[floating[_16Bit] | integer[Any] | numpy.bool[builtins.bool], tuple[Any, ...]] | SequenceND[CanArrayND[floating[_16Bit] | integer[Any] | numpy.bool[builtins.bool], tuple[Any, ...]]] | SequenceND[JustFloat | int]"  [arg-type]
+ tests/special/test_ufuncs.pyi:75: note: "list" is missing following "CanArrayND" protocol member:
+ tests/special/test_ufuncs.pyi:75: note:     __array__

[randolf-scholz]

@jorenham Ugh, terribly sorry but that wasn't sufficient. mypy still resolves _ScalarT to Never in this overload:

@overload
def array(
    object: _ArrayLike[_ScalarT],
    dtype: None = None,
    *,
    copy: bool | _CopyMode | None = True,
    order: _OrderKACF = "K",
    subok: bool = False,
    ndmin: int = 0,
    ndmax: int = 0,
    like: _SupportsArrayFunc | None = None,
) -> NDArray[_ScalarT]: ...

because it doesn't find any constraints on _ScalarT when doing np.array([]), so mypy emits var-annotated¹.

So actually it is this typevar that needs a default (either Any or np.generic I guess).

Again, sorry for not testing this more thoroughly locally, doing the default on _NestedSequence itself in hindisght obviously only helps when we use _NestedSequence directly, but not with def [T](_NestedSequence[T]) -> Array[T] type constructors. Then we need def [T=Any](_NestedSequence[T]) -> Array[T] instead.

Footnotes

1. Which, let's be honest, probably shouldn't even be an error enabled by default. pyrights / pyrefly / ty all use list[Unknown] when doing x = [] and don't emit any errors with standard settings. ↩