BeforeSuite/AfterSuite run even if there are no tests

[SunMar]

Hi,

A few days ago I found some behaviour of which I'm not sure whether it's a bug or a feature. In my behat.yml I have defined multiple suites, and in my contexts I have also defined a @BeforeSuite hook. When I run a test passing the path of the feature file and no --suite= option, Behat will cycle through all my suites executing the @BeforeSuite (and I presume, but I did not test, also the @AfterSuite) for every suite in my behat.yml. Even though via the paths: configuration the feature I was running is only defined for one suite.

If running without --suite, but with a filter (whether it's passing a specific feature file or using something like --tags) I would expect Behat to ignore any suites that don't have any matching features or scenario's. And if a suite is ignored, you also don't need to run any @BeforeSuite and @AfterSuite.

Can this be considered a bug, or it is an intentional feature? For example is it possible to manipulate the filters or paths or something in the @BeforeSuite that might cause Behat to run scenario's for the suite even though initially nothing would have matched?

If this can be considered a bug (or unintentional behaviour) I don't mind diving into the code to see if I can create a PR for it, but I'm not sure where to start looking where something like this can be changed.

[ciaranmcnulty]

Hm, not sure what the correct behaviour here would be

[carlos-granados]

I think this should be considered a bug. I really don't think that Behat should be calling any of the suite hooks if it is not running any feature or scenario in the suite. We probably need to confirm if a similar thing is happening with the feature hooks, if they are being called for features even if no scenario is run. @acoulton what do you think?

[stof]

IIRC, Behat is not eagerly loading the list of scenarios that will be run. It relies on an iterator, which allows it to scale to a huge amount of features without consuming tons of memory (assuming you don't use a formatter that buffers huge amount of memory).
Because of that, by the time the BeforeSuite hook is due to be run, Behat has no idea how many features will be run in the suite (which can be 0). And if we run BeforeSuite for empty runs, it makes sense that AfterSuite also runs.

note that this is also what happens in PHPUnit when you have a testsuite with 0 tests: https://github.com/sebastianbergmann/phpunit/blob/f3d49bb34256407cbcd256aa0e71f1079b4d08b5/tests/end-to-end/event/data-provider-exception.phpt#L23-L29

[carlos-granados]

@stof you are probably right. Let me confirm this and if it is as you explain I'll close this ticket

[acoulton]

If we close this maybe we should just mention the behaviour in the hooks documentation.

[carlos-granados]

Confirmed that with the current implementation it is not possible to know in the beforeSuite hook if there are any tests to run. Changed the type of the issue to documentation so that we can at least document this behaviour