repeating a test spams pytest cache

[dpinol]

When running many repeatitions of a test with --count, .pytest_cache/v/cache/nodeids may get up to millions of entries. Eg:

"test/utils/test_types.py::test_if_none[1-100]",
  "test/utils/test_types.py::test_if_none[10-100]",
  "test/utils/test_types.py::test_if_none[100-100]",
  "test/utils/test_types.py::test_if_none[11-100]",
  "test/utils/test_types.py::test_if_none[12-100]",
...

This may cause a lag of several seconds even after executing a single method with -k specifying a single test file.

The culprit is pytest_sessionfinish in cacheprovider.py, when calling config.cache.set("cache/nodeids", sorted(self.cached_nodeids)). When cached_nodeids contains milions of entries, this function always generates a json with the whole contents of .pytest_cache/v/cache/nodeids.

[RonnyPfannschmidt]

Please elaborate on the actual numbers

[dpinol]

The length of ".pytest_cache/v/cache/nodeids" depends on the usage. I guess I executed at some time with millions of executions and got the file with so many lines. In that case, on my intel i9 a trivial test took 4s, while after removing the file the whole execution now takes 0.3s

[dpinol]

I tried to reproduce it. I executed 1M times a file with 2 tests. It created a cache file of 143M with 2M entries.
Executing the tests a single time passed from 0.4s to 2.4s

[RonnyPfannschmidt]

as repeat uses parameterize - i suspect a breaking change is needed to make this more usable

[okken]

You can disable caching entirely while running large counts with -p no:cacheprovider.
Would that work?

[okken]

Recommend closing this issue, as there's not much this plugin can do about it.

[dpinol]

Sorry for the late answer. -p no:cacheprovider works like a charm, thanks!

I guess users would appreciate a note in the documentation about this issue, probably mentioning the -p no:cacheprovider trick. I added it to my pytest wrapper script when it detects --count since I guess the cache make not much sense when repeating a tests.

[dmakhno]

Is it related to when pytest_generate_tests is executed?
Lmk if my case is another one, and I'll create separate issue.

I have a suite, let's say with 10000 tests.
And I have -k or -m which selects among them only 10.
And e.g. I'm using --count 10
What I'd expected:
"Run 100, Deselected 9990"
What I have:
"Run 100, Deselected 99900" - tests were multiplied before filtering

Is there way to postpone generate hook or "multiplication" call later somehow?