gh-148259: make list.remove atomic for non-trivial __eq__ comparison

[KowalskiThomas]

What is this PR?

This PR addresses the bug reported in #148259 where calling rich comparison may result in releasing the GIL which may result in list.remove removing the wrong item from the list.

The PR also updates the tests added for #126033.

• Issue: list.remove is not atomic for non trivial __eq__ comparisons #148259

[colesbury]

Please add a NEWS entry (https://devguide.python.org/getting-started/pull-request-lifecycle/#how-to-add-a-news-entry) and fix the failing tests.

[picnixz]

The raises parameter is now misleading. Please correct it. In addition, please add tests to list itself. And please also update the docs if they need to.

Overall, I'm not convinced by this change. Mutating the list while doing the comparison is unsafe. We just don't want to crash. But making it raise a RuntimeError does not necessarily makes it better. Otherwise, we need to do it for all other mutable sequences in C to have more or less the same behavior.

cc @rhettinger

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[KowalskiThomas]

The raises parameter is now misleading. Please correct it. In addition, please add tests to list itself. And please also update the docs if they need to.

Yes, I had to stop in the middle of my work earlier -- I didn't plan to leave it in that state, sorry.

Otherwise, we need to do it for all other mutable sequences in C to have more or less the same behavior.

I think some sequences already raise RuntimeError when incompatible things are done to them concurrently, right?

[KowalskiThomas]

I just had the time to give this another look. Updated the tests again and now everything passes I believe.

EDIT: just realised you also asked for tests on list; I'll add them later today!

[picnixz]

I think some sequences already raise RuntimeError when incompatible things are done to them concurrently, right?

Yes but if it is not always consistent. I see this as a garbage-in/garbage-out case. Do not mutate your list while doing other things concurrently. We prefer not raising and leaving it as a GIGO rather than adding complexity and changing existing behaviors.

We do fix cases where there hard crashes (segfaults) but ortherwise we try to retain existing behaviors. Anyway, I think we need first to discuss this, perhaps even standardizing this into a PEP, that is, make sure that concurrent mutations consistently raise whatever collection you are using (not just lists or dicts) and how.

[picnixz]

Yes, I had to stop in the middle of my work earlier -- I didn't plan to leave it in that state, sorry.

In this case, please make it a draft.