Logic in TestSuiteLoader is brittle and causes "Class FooTest not found" even for valid tests in valid filenames

[ondrejmirtes]

Q	A
PHPUnit version	12.4.5
PHP version	8.4.15
Installation Method	Composer

Summary

We occasionally have a weird PHPUnit warning after test suite run has finished. It reported things like:

There were 2 PHPUnit test runner warnings:

1) Class RequireCommitStartedTransactionRuleTest cannot be found in /Users/ondrej/Development/slevomat/tests/build/PHPStan/Rules/Doctrine/RequireCommitStartedTransactionRuleTest.php

2) Class LogTypeEnumFactoryDynamicReturnTypeExtensionTest cannot be found in /Users/ondrej/Development/slevomat/tests/build/PHPStan/Type/Enum/LogTypeEnumFactoryDynamicReturnTypeExtensionTest.php

But the classes exist and are in a correctly named file.

I found the culprit. Some tests cause DI container to start being generated. The logic does new ReflectionClass in all classes therefore it loads them at that point. And that's where we see these warnings.

If the container is already generated then there are no such errors.

The reason why this happens is because of the logic in TestSuiteLoader:

phpunit/src/Runner/TestSuiteLoader.php

Lines 103 to 140 in 3ef346e

	/**
	* @return array<class-string>
	*/
	private function loadSuiteClassFile(string $suiteClassFile): array
	{
	if (isset(self::$fileToClassesMap[$suiteClassFile])) {
	return self::$fileToClassesMap[$suiteClassFile];
	}
	if (self::$declaredClasses === []) {
	self::$declaredClasses = get_declared_classes();
	}
	require_once $suiteClassFile;
	$loadedClasses = array_diff(
	get_declared_classes(),
	self::$declaredClasses,
	);
	foreach ($loadedClasses as $loadedClass) {
	/** @noinspection PhpUnhandledExceptionInspection */
	$class = new ReflectionClass($loadedClass);
	if (!isset(self::$fileToClassesMap[$class->getFileName()])) {
	self::$fileToClassesMap[$class->getFileName()] = [];
	}
	self::$fileToClassesMap[$class->getFileName()][] = $class->getName();
	}
	self::$declaredClasses = get_declared_classes();
	if ($loadedClasses === []) {
	return self::$declaredClasses;
	}
	return $loadedClasses;

If the searched test class is not in $loadedClasses after:

         $loadedClasses = array_diff( 
             get_declared_classes(), 
             self::$declaredClasses, 
         ); 

but the array is non-empty, that's when the problem happens.

I reproduced it in a small repository: https://github.com/ondrejmirtes/phpunit-test-suite-loader-bug

All I needed to trigger it was to do this in the first test that's executed.

    public static function dataProvide(): iterable
    {
        require_once __DIR__ . '/AnotherFile.php';
        return [['foo']];
    }

    #[\PHPUnit\Framework\Attributes\DataProvider('dataProvide')]
    public function testAssertSomethingAndLoadAllClasses(string $foo): void
    {
        self::assertTrue(true);
    }

That makes the output look like this:

$ vendor/bin/phpunit
PHPUnit 12.4.5 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.15
Configuration: /Users/ondrej/Downloads/phpunit-test-suite-loader/phpunit.xml

.                                                                   1 / 1 (100%)

Time: 00:00.001, Memory: 10.00 MB

There was 1 PHPUnit test runner warning:

1) Class SecondTest cannot be found in /Users/ondrej/Downloads/phpunit-test-suite-loader/tests/SecondTest.php

OK, but there were issues!
Tests: 1, Assertions: 1, PHPUnit Warnings: 1.

[ondrejmirtes]

I forgot to mention that I'm also including SecondTest.php in tests/bootstrap.php, to simulate the behaviour.

[sebastianbergmann]

The test loader works as expected. At least, it works as I expect it to, given the assumption I made a long time ago that only PHPUnit's test loader must load source code files that declare test case classes. Therefore, I do not consider this to be a bug per se.

However, I acknowledge the problem you describe and will consider a pull request that solves it, provided it does not break existing functionality.