ci/test_custom_linters.py meta-tests pass vacuously: capsys does not capture in-process pytest.main() output

[adamtheturtle]

Summary

ci/test_custom_linters.py::test_tests_collected_once (and to a lesser extent test_ci_patterns_valid) has been silently broken on main. It calls pytest.main(['--collect-only', ...]) from inside a running pytest test and reads the collected test IDs out of capsys.readouterr().out. The inner pytest.main() installs its own output capture and emits the collect-only listing through that — so capsys in the outer test sees an empty string. Every call to _tests_from_pattern(...) therefore returns set(), tests_to_patterns ends up empty, and both final assertions

assert tests_to_patterns.keys() - all_tests == set()
assert all_tests - tests_to_patterns.keys() == set()

reduce to set() - set() == set() and pass vacuously. The whole meta-test runs in ~1 second on the full suite, which is the giveaway.

Repro

On main (9d7ea250b... or any recent commit), add a single print to _tests_from_pattern:

def _tests_from_pattern(*, ci_pattern, capsys):
    capsys.readouterr()
    pytest.main(args=["-q", "--collect-only", "--disable-warnings", ci_pattern])
    data = capsys.readouterr().out
    tests = {line for line in data.splitlines() if line and "collected in" not in line}
    print(f"PATTERN={ci_pattern!r} got {len(tests)} tests", flush=True)  # <-- add
    return tests

Then:

$ uv run --extra=dev python -m pytest --capture=tee-sys -v -k test_tests_collected_once ci/test_custom_linters.py
ci/test_custom_linters.py::test_tests_collected_once PATTERN='.' got 0 tests
PASSED              [100%]

Every iteration returns 0 tests; the assertion never sees real data.

Impact

The check is supposed to catch:

1. CI matrix patterns that no longer match any test (typos, deleted tests).
2. Tests in the suite that aren't covered by any CI matrix pattern.

Neither has been caught for a while. While investigating an unrelated CI failure on the Python 3.14 branch I switched the helper to a real subprocess.run([sys.executable, '-m', 'pytest', ...]). That immediately surfaced multiple tests that had drifted out of the matrix:

• tests/mock_vws/test_respx_mock_usage.py::* (added in Add respx support for mocking httpx requests #2973)
• tests/mock_vws/test_target_validators.py::*
• docs/source/httpx-example.rst::line:3,column:1

These tests exist in the repo but were never being exercised by the ci-tests matrix.

Suggested fix

Switch _tests_from_pattern (and test_ci_patterns_valid's pytest.main call) to spawn a real subprocess and parse result.stdout/check result.returncode. That isolates state from the outer pytest run (also incidentally side-stepping a pytest-beartype-tests re-wrapping issue under Python 3.14, see pytest-beartype-tests#30).

The Python-3.14 branch already carries this fix in commit fbd452ce / 86d13cbf, but the meta-test breakage and the missing matrix entries are pre-existing on main and worth fixing there independently of the 3.14 work.