Fix failing tests

[swissspidy]

Fixes some tests on WordPress 4.9.

There's also one failing test against WordPress trunk, but not sure why.

Started on the 5th but was fine on the 4th. See https://github.com/wp-cli/extension-command/actions/runs/13644528411
and https://github.com/wp-cli/extension-command/actions/runs/13666785798.

Thought maybe https://core.trac.wordpress.org/changeset/59926 is related. Otherwise couldn't find a suspicious commit.

https://github.com/wp-cli/extension-command/actions/runs/13755011683/job/38461015602#step:11:127

[swissspidy]

Note sure if the remaining 4.9 failures are related to #440. Don't have PHP 7.2 locally right now to test. cc @mrsdizzie

[mrsdizzie]

I was able to set up php 7.2 locally and I can't reproduce any errors when running a copy of WordPress 4.9.26 but when trying to run some of the tests I am seeing these failures:

Notice: An unexpected error occurred. Something may be wrong with WordPress.org or this server&#8217;s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/">support forums</a>. (WordPress could not establish a secure connection to WordPress.org. Please contact your server administrator.) in /private/var/folders/v6/gqkd9_rd74171hsndwtjrxww0000gn/T/wp-cli-test-run--67cf0b401e37f3.85700623/wp-includes/update.php on line 347

It works when running the command against a local WordPress install outside of the testing environment so it isn't a general problem...

[mrsdizzie]

Related, I wonder if it is worth (or possible) to run behat tests in the ci with more detailed output, so when a test fails you get a better idea of why. I always run my local tests like:

composer behat -- features/plugin.feature --format pretty

I know there are other -vvvv type options. Maybe we don't want to fill logs with all of that, but when there is an error it would be helpful to have more than exit status: 0 (Exception). I'm not sure if the local error I've seen is the same one CI is having or not 🤔

[swissspidy]

Sure yeah that's possible.

We could add a change to https://github.com/wp-cli/.github that runs test with the --format=pretty argument when re-running tests with debug logging, see this screenshot:

Related, there's also this long-standing bug to have better output when comparing actual output vs expected output, see wp-cli/wp-cli-tests#17

[mrsdizzie]

oo neat! -- better output would indeed be nice but at least showing the specific error/stack trace is helpful for seeing why something failed.

[swissspidy]

wp-cli/.github#122

[mrsdizzie]

I think my issue is local to my php 7.2 setup somehow, so maybe a red herring. Looking at failures this is probably similarly related to plugins that have since added higher WordPress requirements than 4.9 and generate different output on 4.9 now. I'll try and get php 7.2 working right so I can run the tests locally

see https://wordpress.org/plugins/wordpress-importer/ for example

[swissspidy]

I thought so too but e.g. this result is confusing.

004 Scenario: Create, activate and check plugin status # features/plugin.feature:3
      Then STDOUT should be a table containing rows:   # features/plugin.feature:65
        name	status	update	version	update_version	auto_update	requires	requires_php
        akismet	inactive	unavailable	4.0.1	5.3.7	off	5.8	5.6.20
        hello	inactive	available	1.6	1.7.2	off	4.6	
        Zombieland	active	none	0.1.0		off		
        no-mail	must-use				off		
        polyfills	must-use				off		
         (Exception)

And here's that line in the test:

extension-command/features/plugin.feature

Lines 64 to 67 in 33afe4f

	When I run `wp plugin list`
	Then STDOUT should be a table containing rows:
	| name | status | update | version | update_version | auto_update |
	| Zombieland | active | none | 0.1.0 | | off |

It only tests the line for a plugin that's not in the repo.

But if it's really just about incompatible version we can bump the minimum requirement of those affected tests.

In the meantime, running the tests here with debug logging enabled:

https://github.com/wp-cli/extension-command/actions/runs/13764647737/job/38509224929?pr=443

The extra output isn't really helpful though 🤔

(Note to self: this apparently combines --format progress with --format=pretty so there are some dots here and there 🤦 )

[mrsdizzie]

Maybe since the test is missing the new columns it is failing? Since the test does plugins list and doesn't specify just this one then other plugins trigger the new columns to appear

[swissspidy]

Then it should fail on other versions too, no?

Feels like I really need to try to install 7.2 again 🙃

Edit: got 7.2 installed but having troubles downgrading MySQL because of the changed authentication. Ugh.

[mrsdizzie]

Then it should fail on other versions too, no?

Not necessarily. The way it works is that if wp cli detects an update is unavailable because of wp or PHP requirements, it then automatically displays the requires_wp and requires_php columns (otherwise you wouldn't know why it was unavailable). Since akismet now has a WordPress requirement of 5.8, it is going to produce different table results for the 4.9 test than it does on all our other tests which use 6.1 or higher.

So that is an issue, but the real issue is that this test really only wants to check details for a single plugin, but is running something more general like wp plugin list which now lets other plugins (akismet in this case) alter the output of the table.

So for this particular test, if the goal is to really only check the status of one plugin it should maybe be using wp plugin status Zombieland because the table columns of wp plugin list are now variable depending on what version of WordPress is used since only an older one is going to trigger the new columns to be displayed.

All that being said I'm not sure if that is why the test is failing because I still can't easily get older versions of WP running in tests locally due to wp-cli/wp-cli-tests#243)

[swissspidy]

So for this particular test, if the goal is to really only check the status of one plugin it should maybe be using wp plugin status Zombieland because the table columns of wp plugin list are now variable depending on what version of WordPress is used since only an older one is going to trigger the new columns to be displayed.

Makes sense to me 👍

[codecov]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

[mrsdizzie]

@swissspidy all better ✅

[swissspidy]

@mrsdizzie oh neat, thanks a lot!