Make test suite compatible with PHPUnit 10 / 11

[dingo_d]

PHPUnit 10 was released back in February of 2023, and included ​many breaking changes - they completely rewrote the internal functionality of PHPUnit (mainly the event system for extending the test runner). Besides that, they upped the minimum PHP version to >=8.1, so for WordPress, it could only be used on PHP versions of >=8.1.

While this is not a huge and critical update, since the tests are running fine on PHP versions all the way up to version 8.3, it would be a good idea to start preparing for PHPUnit 10 compatibility.

Besides being up to date, it will allow external testing library maintainers (such as ​wp-browser or ​wp-pest), to use the latest versions of both PHPUnit, or dependencies that are using PHPUnit 10 (such as PestPHP v2 which depends on PHPUnit 10).

I understand that this is a huge undertaking, but it's okay to have a ticket open for this (I didn't find one opened while searching, could be wrong tho), so that we can keep track of changes, and maybe create a roadmap or a to-do list of what needs to be fixed in order to make WordPress test suite compatible with PHPUnit 10.

[desrosj]

I went through and made a preliminary list by looking at the ​removed section within the 10.0 changelog and searching the test directory for occurrences of removed elements. The following have been removed and are found in the test directory:

• withConsecutive() (​details)
• MockBuilder::setMethods() and MockBuilder::setMethodsExcept() (​details)
• expectDeprecation(), expectDeprecationMessage(), and expectDeprecationMessageMatches() methods
• expectError(), expectErrorMessage(), and expectErrorMessageMatches() methods
• expectNotice(), expectNoticeMessage(), and expectNoticeMessageMatches() methods
• expectWarning(), expectWarningMessage(), and expectWarningMessageMatches() methods

This is by no means an all inclusive list. But wanted to provide a starting point if someone wanted to try and tackle this.

To run the test suite, the yoast/phpunit-polyfills version constraint needs to be updated to ^2.0.0 in the composer.json file.

[ironprogrammer]

To provide more detail to the list from comment:1, here are the deprecated expect*() warnings to increase discoverability of this ticket (​details):

Expecting E_DEPRECATED and E_USER_DEPRECATED is deprecated and will no longer be possible in PHPUnit 10.
Expecting E_ERROR and E_USER_ERROR is deprecated and will no longer be possible in PHPUnit 10.
Expecting E_STRICT, E_NOTICE, and E_USER_NOTICE is deprecated and will no longer be possible in PHPUnit 10.
Expecting E_WARNING and E_USER_WARNING is deprecated and will no longer be possible in PHPUnit 10.

Props @azaozz for pointing this out.

[desrosj]

Related: #60305.

[swissspidy]

#60551 was marked as a duplicate.

[jujuguru]

Adding my experience here with a workaround.

WP_UnitTestCase_Base::expectDeprecated makes a call to \PHPUnit\Util\Test::parseTestMethodAnnotations during test setup and renders testing useless.

My simple workaround
I've commented the call to parseTestMethodAnnotations out in the file abstract-testcase.php and it has allowed test to complete.

I will caveat that I'm only early doors with testing so other issues may arise. If they do and I figure out workarounds I'll share.

Current set up:
PHP 8.2.15
PHPUnit 11.0.3
WordPress 6.4.3
Yoast Polyfills ^2.0

[jrf]

Just found this ticket. There is much bigger problem we have to deal with before this could be contemplated (complete reorganization of the test suite). I don't think adding my analysis in a comment in this ticket is the right way forward, so I propose I open a new ticket with a detailed analysis and roadmap to replace this ticket.

[chesio]

Replying to jujuguru:

My simple workaround
I've commented the call to parseTestMethodAnnotations out in the file abstract-testcase.php and it has allowed test to complete.

Thanks for the hint, I hit the issue too while running integration tests of one of my plugins. I use a slightly more flexible workaround:

<?php

namespace My\Plugin\Tests\Integration;

/**
 * Base class for all integration tests
 */
abstract class TestCase extends \WP_UnitTestCase
{
    /**
     * @internal Workaround to allow the tests to run on PHPUnit 10.
     *
     * @link https://core.trac.wordpress.org/ticket/59486
     */
    public function expectDeprecated(): void
    {
        return;
    }
}

This way WordPress test suite can be kept up to date without further edits.

[outsellers]

Replying to jujuguru:

Adding my experience here with a workaround.

WP_UnitTestCase_Base::expectDeprecated makes a call to \PHPUnit\Util\Test::parseTestMethodAnnotations during test setup and renders testing useless.
My simple workaround
I've commented the call to parseTestMethodAnnotations out in the file abstract-testcase.php and it has allowed test to complete.

I can attest that this also worked for myself. For the time being, no issues after commenting this out.

[hellofromTonya]

Status Update:

Thank you @vphrudy for the ping in the Making WordPress #core slack channel.

As @jrf noted:

There is much bigger problem we have to deal with before this could be contemplated (complete reorganization of the test suite). I don't think adding my analysis in a comment in this ticket is the right way forward, so I propose I open a new ticket with a detailed analysis and roadmap to replace this ticket.

​Juliette has drafted an epic (bigger picture proposal) ticket for this initiative. The initiative is currently blocked by solving the "deprecation expectation thingie":

(and yes, the deprecation expectation thingie is the blocker for the release - I'm still trying to figure out a way to polyfill that, though 3.0 might just end up not supporting PHPUnit 10 (which would bypass the problem))

Getting WordPress Core running on PHPUnit 11.1 will require significant changes and challenges to solve:

• Core's entire test suite will require rearchitecting and reorganization (reasonings will be shared in the epic ticket).
• Handling of "deprecation expectation thingie" is a hard challenge, one that will be solved as part of PHPUnit Polyfills 3.0 (that will have PHPUnit 11 support).

The current plan:

• Close this ticket in favor of the new epic ticket (I added close keyword).
• The new epic ticket will track the bigger picture of the proposal and initiative. Individual tasks will then have their own tickets for tracking their specific (smaller) scope of work and discussions.
• WP will support PHPUnit 8, 9 and 11.1+, skipping PHPUnit 10 and 11.0 (reasons will be shared in the epic ticket).
• Tests suite's rearchitecture and reorganization task will be tracked in its own ticket and can happen in tandem. (Early work PRs are available, though waiting on the proposal before moving them into a ticket and out of draft.)

[dingo_d]

I'll close the ticket, and it can be tracked in the epic as suggested.